feat(prqlc)!: preprocess Jinja templates

[aljazerzen]

See demo repo: https://github.com/aljazerzen/jaffle_shop

Method

We start with Jinja-PRQL:

from {{ ref('my_model') }}

... parse Jinja and extract interpolations:

from _jinja_0

... compile PRQL:

SELECT * FROM _jinja_0

... replace anchor with original Jinja expressions:

SELECT * FROM {{ ref('my_model') }}

Minijinja

I used minijinja to lex the input into tokens. For that I had to fork the project so it exposes the lexer.

Problems

This approach does not work any jinja expression that would not produce exactly one output token. This includes for loops and if statements.

We could do full Jinja evaluation and provide all functions that are available in dbt, but set them to evaluate to anchor tokens just as I do now. This would make for loops work, but could not fully emulate evaluation that depends on any state that is provided by dbt.

I don't know much about dbt, should I'd ask for people to check it out and try to translate a few dbt projects.

Ideal solution

Have access to dbt state, i.e. compile PRQL after dbt evaluates the Jinja template.

[aljazerzen]

Because of mentioned limitations, I'm leaning toward not merging this. If anyone does find any value in using this workflow on dbt projects, we can merge.

[max-sixty]

I admittedly share your perspective re jinja.

Ideally we won't need jinja, because we have a structured language. That's also the perspective of the dbt folks.

Maybe this could be an advanced feature of s-strings? But outside of that, I think we should hold the language itself to the standard of producing the abstractions...

[aljazerzen]

But because prql-compiler does not (and should not) know how to handle Jinja, ...

With this I meant not only for-loops and other evaluation, but even parsing Jinja templates.

from {{ ref('table') }}

... should be syntactically invalid PRQL code

[max-sixty]

... should be syntactically invalid PRQL code

Oh yes I agree; we can remove Jinja, either now or when we get the proper dbt plugin up.

IIRC, in the end we don't even use it in dbt-prql — PRQL actually gets called after dbt has processed the jinja (which is why it doesn't work with all DBs). Though my memory is hazy and I'd need to confirm, depending on how quickly you think we should remove it!

[max-sixty]

Let's discuss this on the next dev call. In some ways I think this is a very clever implementation. OTOH it does require everyone using it to have prqlc, which is a big disadvantage for teams, and have prqlc watch running.

It is also quite a nice demo of prqlc watch!

[max-sixty]

Should we remove it from the pest grammar?

[max-sixty]

@aljazerzen we should release this as soon as possible; as at the moment we can't release PRQL...