Shellscript Syntax Highlighting never recognizes the end of a <<-EOF delimiter

[nathan-thomas]

Prerequisites

• [X ] Put an X between the brackets on this line if you have done all of the following:
  • Reproduced the problem in Safe Mode: http://flight-manual.atom.io/hacking-atom/sections/debugging/#using-safe-mode
  • Followed all applicable steps in the debugging guide: http://flight-manual.atom.io/hacking-atom/sections/debugging/
  • Checked the FAQs on the message board for common solutions: https://discuss.atom.io/c/faq
  • Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aatom
  • Checked that there is not already an Atom package that provides the described functionality: https://atom.io/packages

Description

As I'm sure you know, you can throw a bunch of text into a file by using something like:

cat > $1 << End-of-File
text here
more text
End-of-File

I'm presently doing just this in a script, inside a function and have indented whitespace:

cat > $1 << End-of-File
    text here
    more text
End-of-File

If you do this, you'll end up with the whitespace in your output file. However, you can ignore that whitespace by putting a dash (-) before your End-of-File delimiter. Example: cat > $1 <<-End-of-File

The issue is that when I include this dash, syntax highlighting for this script breaks from then on, because the language-shellscript interprets everything after <<-End-of-File as text that is going into the file, which isn't correct.

Steps to Reproduce

1. Copy this code into Atom and invoke the Shellscript Language package:

cat > $1 << End-of-File
    text here
    more text
End-of-File

2. Insert a dash before End-of-File (or simply replace the first line with the following) and see the difference:
   cat > $1 <<-End-of-File

Expected behavior: [What you expect to happen]
At the end of the file, I expect the rest of the script to revert to regular syntax highlighting.

Actual behavior: [What actually happens]
The Language package interprets the rest of the script as being a part of the cat statement.

Reproduces how often: [What percentage of the time does it reproduce?]
100%

Versions

Atom : 1.15.0
Electron: 1.3.13
Chrome : 52.0.2743.82
Node : 6.5.0

apm 1.15.3
npm 3.10.5
node 4.4.5 x64
python
git 2.12.0.windows.1
visual studio

I am running Windows 10.
You can get this information from copy and pasting the output of atom --version and apm --version from the command line. Also, please include the OS and what version of the OS you're running.

Additional Information

I haven't checked whether this is happening in other operating systems.

Any additional information, configuration or data that might be necessary to reproduce the issue.

[winstliu]

Fixed by #77.

[baziliolg]

Looks like I still have similar issue with Atom 1.21.0.
This code snippet is rendered incorrectly.

function blabla() {
    local _s3tag=`python - << END
import sys, xml.etree.ElementTree as ET
p = ET.parse("${CODE}/blabla.xml").findall('property')
for i in p:
    if i.get('name') == "s3tag":
        print i.get('value')
END`
    # Here we should have normal syntax highlighting but instead it is all red
    [[ -n ${_s3tag} ]] && echo "Found _s3tag: ${_s3tag}"
}

Screenshot link

[winstliu]

@baziliolg That issue is very hard to fix with the current TextMate-based grammar system. I'm sure you know that heredoc identifiers must end at the end of the line, and so that's what language-shellscript requires. However, since it's in an interpolated string, the "end" of the line is actually the `. Unfortunately, there's no way to tell the heredoc match that without substantial pattern duplication.

[ryran]

I think there's been a regression. I'm not sure when it started, since I've been doing pretty much only python for quite a few months, but I just opened a shell script in atom-1.32.0-beta0 and immediately noticed the syntax highlighting was off.

Here's a simple reproducer.

In case it's not clear, the syntax highlighter didn't catch the EOF with tab before it. The following shows what it should look like (though of course the hereword shouldn't have to be left-justified).

[Arcanemagus]

@ryran It is actually a bug in the TextMate grammar that the tab is allowed before EOF, not the other way around 😉.

According to the Here Documents reference documentation the closing limit string must start at the first character of a line.

As to why this changed: The Tree-sitter parsing system was recently enabled by default in the v1.32 betas, as announced. You can switch back to the TextMate parser by disabling the option, but again, your specific scenario is now correct and was previously incorrect.

[ryran]

Thanks for the response @Arcanemagus

According to the Here Documents reference documentation the closing limit string must start at the first character of a line.

Uhhh ... (1) the Advanced Scripting Guide is not "reference documentation" and (2) it fails to mention anything explicit about where the limit string needs to be when redirection operator is <<-.

In contrast, the BASH man page makes it crystal clear:

$ man bash | grep -A2 'If the redirection operator is <<-'
   If the redirection operator is <<-, then all leading tab characters are stripped
   from input lines and the line containing delimiter. This allows here-documents
   within shell scripts to be indented in a natural fashion.

Note that this is a very common idiom for bash scripts. The following script works just fine:

#!/bin/bash
usage() {
	cat <<-EOF
		Usage: ${0##*/} FOO BAR
	EOF
}
[[ $# -eq 2 ]] && echo $2 $1 || usage

However, notice that Github's colorization only sees a problem with the following (where I removed the - from the <<-).

#!/bin/bash
usage() {
	cat <<EOF
		Usage: ${0##*/} FOO BAR
	EOF
}
[[ $# -eq 2 ]] && echo $2 $1 || usage

And of course it fails when I try to run it.

$ ./x.sh  a b
./x.sh: line 8: warning: here-document at line 3 delimited by end-of-file (wanted `EOF')
./x.sh: line 9: syntax error: unexpected end of file

[Arcanemagus]

Please file a new issue with that information then so this can get fixed, thanks!

[ryran]

report moved to new #127