Heredoc formatting in the bash lexer is not correct

[moesasji]

Currently vis has a set of patches contributed by @silasdb aiming to improve the formatting of heredocs: fix ending delimiter and a set to recognize <<- here and here.

However even with these patches the formatting is not yet correct for all correct usages of heredoc as it doesn't deal correctly with spaces between << and the delimiter. Moreover the formatting fails when the first delimiter is being quoted to prevent variable expansion. As a result fixing this appears not that trivial.

I've attached a test-file that highlights the above problems as this is easier to show than explain in part based on this tutorial that are worth fixing and probably a useful step to get this working correctly.

File including valid usage of heredoc: test-bash.zip

[silasdb]

This is something I'd like to tackle. Before trying your test, I tried to run the Scintillua test suite and I got a bunch of errors. It seems that lexers cannot find the global lpeg (e.g.: `Failed! lexers/rhtml.lua:6: attempt to index a nil value (global 'lpeg')`). Tests work fine if I explictly `local lpeg = require('lpeg')` for each lexer. Could it be my environment? I'm using NetBSD 9.1 Lua 5.3 + LPEG. Not sure if you have this problem in GNU/Linux or another *nix.

[moesasji]

I get the same error message on Slackware Current if I use lua 5.3.
The tests complete without error if I use lua 5.1 (still the default one)

[orbitalquark]

*tests.lua* should probably define `_G.lpeg`. My computer is set up to have lua5.1 execute the tests and I haven't bothered to run it with any other lua version.

[silasdb]

I've worked on small changes in a branch of mine that can be found here: https://github.com/silasdb/scintillua/tree/s-0-bash Comparison against default branch can be seen here: silasdb@default...silasdb:s-0-bash In summary, it handles all cases listed in test_bash.zip that moesasji attached earlier, with exception of delimiters that start with backslash (this will be fixed in a next commit). The pattern expression became too complex. Separating functions for "<<" and "<<-" wil not make the expression simplier, but will make branch (if .. then) unecessary. I just want to ask you if you have a better idea before I resume the work. As expected, if a pipe or redirection shows up just after the starting delimiter, it is highlighted as part of the heredoc: cat << EOF > file.txt The current working directory is: $PWD You are logged in as: $(whoami) EOF @moesasji, another issue: the example you attached has ending delimiters with trailing spaces. The ending delimiter is handled by bash (and the classic Bourne shell) literally, which means that, in this context, a literal space is necessary for the starting delimiter. Try: cat <<EOF foo EOF # the line above ends with a space It will fail. To make it work, remove the trailing space at the ending delimiter or write the following is necessary: cat <<EOF\ foo EOF # both the starting and the ending delimiter end with a trailing space I've attached [test_bash2.zip](https://github.com/orbitalquark/scintillua/files/5853207/test_bash2.zip) that is the same file as test_bash.zip, but with examples changed to not have trailing spaces. Currently, my solution cannot parse the case with trailing spaces (not sure if it is possible with current capabilities of Scintilla/Scintillua).

[moesasji]

@moesasji, another issue: the example you attached has ending delimiters with trailing spaces. The ending delimiter is handled by bash (and the classic Bourne shell) literally, which means that, in this context, a literal space is necessary for the starting delimiter.

Many thanks for working on this @silasdb; the trailing space example is probably just a left-over from copy/pasting the examples from random websites as I couldn't find clean info on what exactly the grammar for HEREDOC is. It is not something I spotted as Vim mostly handled the cases correctly.

As having such trailing whitespaces be relevant for the delimiters is generally a very bad idea it is probably indeed best to just remove that trailing whitespace from my test-example as you did.

[orbitalquark]

I'm going to close this, as the changes in 956e4ce pass all but the "\EOF" test in the test zip.