Support real heredocs (and rename triple quote syntax)

[ingydotnet]

Retitled this issue. Triple quotes are not heredocs. They should be called triple quotes as other languages do. Also it would be fantastic to have real heredocs in CoffeeScript such that literal chunks of programming related texts could be coded.

Original post below:

---

This gist https://gist.github.com/6b91be7ec9c623025782 shows how Bash, Perl and Ruby do single-quoted heredocs vs coffee.

With heredocs you want to be able to have a block of any untouched text. (Only the terminator is special). Double quote semantics are very useful, but single quoted ones, much less so.

I found this trying to load a yaml doc like this: https://gist.github.com/185217222594a161bada

You can easily imagine wanting to heredoc a block of coffeescript, or many other syntaxes that include backslashes.

[ingydotnet]

FWIW, I'm not sure that CS thinks off these Pythonesque triple-quotes, as heredocs, but I saw them called that in the changelogs.

https://gist.github.com/anonymous/8c6a46b225256b5542b9 would seem to be problematic.

[xixixao]

This is JavaScript, you need to escape backslashes. Would you disallow \n then and just keep everything as is?

[ingydotnet]

I'm not sure how to respond except with more examples. Of course you need to compile to valid JS (where all backslashes in quoted strings are escaped) but the syntax you use to get there is negotiable.

Here is some CoffeeScript:

grammar = yaml.load '''
  hr:
match: ----\\n
  head:
match: (={1,4}) (.*?)( =+)?\\n
  para:
match: ((?:(?!----\\n)\\S.*\\n)+)
  blank:
match: \\n
'''

Here is the generated JavaScript:

// Generated by CoffeeScript 1.6.3
var grammar;

grammar = yaml.load('hr:\n  match: ----\\n\nhead:\n  match: (={1,4}) (.*?)( =+)?\\n\npara:\n  match: ((?:(?!----\\n)\\S.*\\n)+)\nblank:\n  match: \\n');

I want the exact same JS, with this CS:

grammar = yaml.load '''
hr:
  match: ----\n
head:
  match: (={1,4}) (.*?)( =+)?\n
para:
  match: ((?:(?!----\n)\S.*\n)+)
blank:
  match: \n
'''

which I could do in any language's heredocs except CoffeeScript. Heredocs in all the languages that support them have 2 semantics: 'with escaping' and 'with none'. They all use single quote syntax to indicate the no escaping semantic. eg <<'EOS'. Even YAML has literal blocks, so that any possible YAML text can be written as a string inside with no additional escaping (just indentation):

a yaml document as a value: |
  regex: stars \**
  even a literal: |
    foo: bar
foo: bar

Here's an example using CoffeeScript itself:

console.log "Here is an example CoffeeScript program:\n"
console.log '''
  for [1..3]
    console.log "Hello, world\\n"
  '''

Note you have to needlessly escape the \. Imagine the example heredoc wanted to have a heredoc in it!

The CoffeeScript syntax of triple quotes is taken from Python, where that syntax is not called a heredoc, and the semantics are not those of a heredoc.

From Wikipedia: "In computer science, a here document (here-document, heredoc, hereis, here-string or here-script) is a file literal or input stream literal: it is a section of a source code file that is treated as if it were a separate file.". ( http://en.wikipedia.org/wiki/Here_document )

In other words, the text in the heredoc should be exactly what would be read from a file. In fact, in Shell, heredocs act as files, rather than strings (you can cat a heredoc).

I'm not certain that the ''' syntax can be changed to support heredoc semantics at this point. I'm fairly certain that it shouldn't be called a heredoc. I am very certain that I would love to have heredoc semantics in CoffeeScript.

[ingydotnet]

@xixixao: I think I misread your query. To answer it more tersely: Yes.

Yes, heredocs should treat \n as "\n" (in the "no escaping" semantic) and every character until the terminator should be asis.

Triple quotes have the problem that they usually can't have unescaped triple quotes in them. You can get around that easily by lexing for a line of the appropriate indentation that contains only a triple quote. CoffeeScript's lexer doesn't seem to do that now.

I doubt that changing the current semantics of triple quotes would be wise, since it would introduce subtle changes to existing code. That said, I find it hard to believe that the lack of heredoc semantics has gone unnoticed so far. (Especially in something purporting to be a heredoc).

The traditional starting tokens: <<EOS, <<'EOS', <<"EOS" etc are not really sexy enough for CoffeeScript syntax, although they do have well established meanings.

Thoughts?

[bjmiller]

That's not the only use for triple-quotes. This is also valid (and, I use it more often):

"""<a href="#{link}">The Link</a>"""

Just scanning for a bare set of triple-quotes doesn't cover that.

[ingydotnet]

I meant to imply something like this should work:

html = """
  <a href="#{link}">The Link</a>
  <!-- and you can even have triple quotes
  """<a href="#{link}">The Link</a>"""
  -->
  """

This does not mess with your use case as the opening and closing tokens are on their own lines. As if it were:

html = <<"EOS"
  <a href="#{link}">The Link</a>
  <!-- and you can even have triple quotes
  """<a href="#{link}">The Link</a>"""
  -->
  EOS

While making that example I think I found 2 more issues with heredocs, but will file as separate issues.

[xixixao]

@ingydotnet Could you edit the top post so that it's clear the issue lies with escaping. Also, your proposal is to add true heredocs and rename current single quoted heredocs?

In regards to using single quoted heredocs without escaping, seems that ruby simply allows you to escape single quotes (and I guess breaks your ideal heredoc behavior):

print ''' ahoj \'\'\' \n asd ''' # ahoj ''' \n asd

If your use case is to be able to put source code (possibly including heredocs) into heredocs, it seems that you always have to do some escaping (in which case, escaping backslashes might be as difficult as escaping single quotes).

[lydell]

Possibly a bit related: #2493

[ingydotnet]

@xixixao Your Ruby example is "triple quotes", not "single quote heredocs", which would be:

print <<'eos'
ahoj \'\'\' \n asd
eos
#ahoj \'\'\' \n asd
#

@lydell yes, that's related. I can think of many more use case examples: YAML, JSON, templates, CoffeeScript, JavaScript, Markdown, etc, etc

[GeoffreyBooth]

Heredocs would be cool. The docs don’t call triple-quoted strings heredocs anymore (though the word does appear, erroneously in the changelog; not sure that’s worth revising).

Closing as this isn’t a top priority at the moment, but a PR that implements heredocs would be welcomed.

[johndeighan]

If I get the chance, I might attempt to create that PR (I'm pretty busy right now). Meanwhile, here's an example of what you can do with real HEREDOC syntax aka Perl, that you can't do with triple quoted strings:

hVars = {name: 'John', age: 68}
assert.equal(process(<<"", hVars), <<"", "variables in square brackets should be replaced")
This is a junk line
Hi, my name is [name] and I'm [age] years old

This is a junk line
Hi, my name is John and I'm 68 years old

Note how the line containing the assert is a complete statement all on one line - no need to scan down to past the strings to see what's being compared, what the name of the test is, etc. Also, FYI, I prefer to terminate each HEREDOC string with a blank line (works best if you set up your text editor to always strip all trailing whitespace). Much easier to grok.