feat: Allow s & f strings to contain expressions

[max-sixty]

This doesn't work at all, but using it to ask a question of the best way of doing #3279 — how can we parse an expression from within another parser?

Here's the relevant line: https://github.com/PRQL/prql/compare/main...max-sixty:prql:try-s-string-expr?expand=1#diff-f3825716552312a9ef9f74ba3196b570761c8907b541923cdd5f6f4dcebb677fR162-R164

[max-sixty]

I think this isn't trivial, but I might be over-complicating it:

• Currently the Interpolation Token contains a String. That would need to change to contain a Vec<Token> https://github.com/prql/prql/blob/4bb4043aec6ad7259dd8572c7a8fd4a81f459789/crates/prql-parser/src/lexer.rs#L17
• That means the lexer needs to become recursive, so it can be passed into the interpolation lexer at https://github.com/prql/prql/blob/4bb4043aec6ad7259dd8572c7a8fd4a81f459789/crates/prql-parser/src/lexer.rs#L82
• Then the interpolation parser would parse Tokens rather than chars (at https://github.com/prql/prql/blob/7a59f1951c6c9259775000a47d6205f8b37bd893/crates/prql-parser/src/interpolation.rs#L106)
• It would get passed the (already recursive) expr parser at https://github.com/prql/prql/blob/04b5174cba01bcc79837e69f0697486210d3d6dd/crates/prql-parser/src/expr.rs#L86, and use that to parse its contents (or at least everything left of :)

That's not a great road to go down and then realize it was all wrong, so I'll try and not proceed until someone else thinks that's reasonable...

[aljazerzen]

I don't quite remember how this works, but I do remember that I wanted to make lexer recursive, but later realized that it is really not a good idea.

[max-sixty]

I wanted to make lexer recursive, but later realized that it is really not a good idea.

OK interesting. Is this just a general thing that it's better to not have recursive lexers, or something specific?

If we want to parse expressions within strings, I guess we do need some sort of thing that resembles recursion, though there are a few possible forms.

[aljazerzen]

That's because lexer a usually limited to operate on regular languages (i.e. whatever can be matched with a regular expression). This limitation is deliberate to limit the complexity of the code for the lexer and leave the recursive stuff to the parser.

[max-sixty]

Yes, though if we want something like python f-strings, is there any way around having recursion in the lexer (or something that is functionally similar to recursion)? I had thought not...

[max-sixty]

As discussed on the dev call — no one is opposed to on principle, but @aljazerzen thinks it's not easy, so we should only do this if it's tractable