Concepts for HEREDOC parsing

[stbuehler]

I couldn't find any note on HEREDOC handling, and also don't see how a reasonable grammar could be looking.

So generally this is about a special syntax that uses a variable "token" that is used to mark the end of a (mostly) unparsed section in the input (if it has to be parsed for string interpolation or similar that could be done in a separate step if needed).

The section usually starts after a new line, but the (start) token doesn't have to be last thing on the previous line (the wikipedia page even has an example for bash: cat << EOF > ~/testFile001), and some languages (e.g. puppet) even support starting multiple HEREDOCs on the same line.

If I had to make a suggestion how to implement this, I'd put in somewhere in the "trivia"-handling - check for HEREDOC context after a newline (which somehow needs to emit the tokens for it). Stack might be helpful, although for multiple HEREDOCs you'd actually want FIFO, not LIFO (i.e. PEEK_FIRST and POP_FIRST), and you'd need a "is the stack non-empty"-match, as otherwise the parser would have to match a lot of content just to realize in the end there is no token for the end of a HEREDOC. And multiple stacks/queues might be helpful.

If there already is a proper way (or perhaps a way to hook into the parser) an example might be helpful.

(I get that probably nothing about this is "elegant" from a parser point of view, so "we don't want to support that" is a reasonable answer, but it'd be nice to know that.)

[tomtau]

I think it might be tricky to nail this kind of dynamic terminator in a single-pass grammar. So, I'm guessing the best approach would be a two-phase parsing setup: in the first phase, it'd capture heredoc IDs/openers and pre-process them to get the content / positions of the corresponding sections. Then, it'd parse the actual content sections in the second phase?