How can dsl::list and expression_production be combined and still handle trailing newlines?

[sarcasticnature]

I am currently struggling to parse a sequence of Expressions separated by newlines and am hoping to get some help.

The basic premise is to parse a 'list' of expressions, separated by newlines, into a vector. I can parse a single line without issue, but when I try to add multiple lines I run into problems with a trailing newline at the end of the file. There are multiple ways to approach the problem, but the few I have tried have failed.

I have lost track of what all I have tried, but out of everything I have found the following two are the most puzzling

1. dsl::list(dsl::p<Expr>, dsl::sep(dsl::ascii::newline)) + dsl::eol + dsl::eof
   • I would think that this would match a file with newline-separated expressions, a trailing newline, and then EOF, but it does not appear to
   • dsl::newline does not seem to do the trick either (in place of dsl::eol)
2. dsl::list(dsl::p<Expr>, dsl::trailing_sep(dsl::newline));
   • This fails to compile, with an error complaining "list() without a trailing separator requires a branch condition"
   • A similar structure using something other than a lexy::expression_production inside the dsl::p does work, however

Here are two playgrounds that exhibit this behavior.
a playground with (1) from above that fails to work:
https://lexy.foonathan.net/playground/?id=3aYM8cMea&mode=tree

a playground with a trailing_sep + list combo (that seems to work:
https://lexy.foonathan.net/playground/?id=TKqrz9YEG&mode=tree

Honorable mentions: I tried using dsl::terminator as well, but it seemed to be the wrong tool for the job (I could be wrong though).

---

Any suggestions on where to go from here? I'm sure I'm missing something simple. Note that in an ideal world I could parse either a single line or multiple lines with the same setup, but I'd settle for two separate structs if I must.

I have tried to distill this problem down as much as possible to make it easier to debug, but I'm happy to give some more details on my particular application if it's relevant/helpful to finding a solution.

[foonathan]

dsl::list(dsl::p<Expr>, dsl::sep(dsl::ascii::newline)) + dsl::eol + dsl::eof

The issue here is that dsl::list with dsl::sep uses the existence of the separator to determine whether it should parse another Expr. So you can't have a newline after the expression.

dsl::list(dsl::p<Expr>, dsl::trailing_sep(dsl::newline));

dsl::list with dsl::trailing_sep can't use the existence of the separator to check whether it can parse another Expr (since it could be the trailing separator), so Expr needs a branch condition to determine whether or not one matches, as the error message told you. Depending on what Expr parses, something like dsl::peek(foo) >> dsl::p<Expr> as list item might do the tricker.

However:

I tried using dsl::terminator as well, but it seemed to be the wrong tool for the job (I could be wrong though).'

No, this is the exact correct tool for the job. It can use the existence of a terminator to determine whether it needs to continue parsing the list.

Fundamentally, it makes sense to rephrase the question: You're not attempting to parse a list of expressions separated by newlines, you're parsing a list of lines, where each line is dsl::p<Expr> + dsl::eol. This list of lines ends when you have EOF: https://lexy.foonathan.net/playground/?id=xK5xsnWa4&mode=tree

(Alternatively, you could also use .list(dsl::p<Expr>, dsl::trailing_sep(dsl::eol)), but I think the other formulation is clearer.)

If in the real example, you have a different terminator, you need to adapt it accordingly. dsl::brackets might also be convenient there as well.

[sarcasticnature]

ah, I must have had a typo or something when I went through some variations of dsl::terminator -- it works as expected now.

Thanks for the help!