Custom block tags

[jameysharp]

Jinja2 supports extensions that implement custom block tags. I'd like to use minijinja to render existing templates that use extension tags. But at least for my purposes there are several good-enough alternatives that are less complicated than Jinja2's full extensible parsing.

The tags I currently care about just minify their rendered contents in various ways, so for a first cut it's sufficient to just ignore the tags and pass through the contents unminified. My plan in the absence of any support from this library is to search for the relevant tags in the template source text and delete them before passing them to minijinja.

But I wondered if it's worth discussing various ways minijinja could provide more support for extensions. Here are a few options that I think would work for me and might help others:

• Allow associating a pair of block tags (e.g. compress/endcompress or spaceless/endspaceless) with a filter that implements the intended string->string transformation on the rendered contents between the two tags. This doesn't require fully general extensible parsing but still supports more unmodified Jinja2 templates.

• Make the lexer public and allow preprocessing the stream of tokens before they're parsed. This allows embedders to implement the above filter-based transformation themselves, but fixes some implementation details as public API.

• Or, of course, make the full AST public and provide all the hooks Jinja2 extensions have today. I'm guessing it's too soon in minijinja's development to commit to that much public API, but I'm including it for completeness.

[mitsuhiko]

Refs discussion #178

[mitsuhiko]

Exposing the lexer/parser/codegen internals I really do not like because it blows up the complexity of the API surface greatly. I think a potential solution would be to add some sort of preprocessing step but even there I'm somewhat skeptical at the moment that this is particularly reasonable.

The lexer is somewhat stable so exposing that is an option, however because of the span information that is carried, adding tokens that do not exist in the source causes all kinds of oddities later. MiniJinja for error reporting purposes assumes that every token exists in the source stream.

I understand that some templates already exist but is {% spaceless %}...{% endspaceless %} really much of an improvement over {% filter spaceless %}...{% endfilter %}?

I already did not like that people made these custom syntax extensions in Jinja2, but unfortunately it really seems like that is almost impossible to fight :-/

[mitsuhiko]

I think that resolving #135 might move this forward. Once the lifetime is gone it would be super trivial to add all kinds of preprocessing steps.

[jameysharp]

For examples like spaceless, I agree completely that a filter is a better choice. I've already made that change in the templates for my project.

https://django-compressor.readthedocs.io/ is an interesting example though. While a filter is potentially a reasonable alternative for its runtime behavior, it also needs to be able to find its own tags in templates outside of the rendering path in order to do offline rendering. For that purpose, having a dedicated AST node is helpful. Jinja2's built-in i18n extension is similar, in that message catalog extraction uses the AST.

Block caching is another example: for it to be useful, it has to defer rendering the subtree until after it discovers a cache miss. A filter isn't enough there. I think that could be built using the call tag though, right? I'm not sure how to save the rendered fragment to the cache in that case. It seems a bit tedious at least without custom tags.

I didn't realize there was a discussion about this already—I'm not used to looking in that section instead of the Issues tab. The "dbt" project example from #178 is interesting because the custom tags save rendered fragments to global state, but I think the doc tag is functionally equivalent to the macro tag.

Looking through GitHub Code Search results, I see Jinja2 extensions have been used for a lot of purposes where they weren't strictly necessary. I understand wanting to avoid that with minijinja. I think there are two kinds of uses that are worth discussing:

1. Tags which change how their subtrees are evaluated, like block caching or trans/pluralize. This isn't a great use case because the base Jinja2 language already has pretty comprehensive control flow primitives, but there might be something to learn there.
2. Tags which annotate parts of a template for external tools, like compress and trans, or similarly for special function calls like _/gettext.

It might help to take inspiration from the ecosystem around Rust's proc-macros, where rustc's internal AST is not exposed, but there's a separate syn crate with a stable API that's useful specifically for programs that want to manipulate or generate Rust source. So the compiler can use any convenient representation and evolve quickly, but nobody has to write a parser/printer from scratch either.

I think enabling the same kind of manipulation of Jinja2 source text might cover all the use cases I can think of. Folks concerned about performance could invoke preprocessors from a build.rs script if necessary. If performance doesn't matter, folks can construct a new heap-allocated String before calling add_template, although for debugging purposes it's nice to be able to pass spans through from the original source, so some level of lexer integration could still be helpful.

[mitsuhiko]

The approach for syn is what I have in mind. However the big blocker for this today are the 'source lifetimes. Unless you are using the Source feature you cannot have an intermediary step today. For that the interface of most things would have to turn from &'source str into Cow<'source, str> so that some preprocessing can put the modified input stream somewhere.

This change is absolutely possible but it's quite complex. So #135 is a very likely blocker that needs resolving first.

[esoterra]

Just wanted to add on that I'd like to use custom block tags to implement syntax highlighting using syntect.

[mitsuhiko]

@esoterra why can you not use the {% call %} block for that?

{% call syntect('python') %}
...
{% endcall %}

or similar.

[mitsuhiko]

I added an example here: https://github.com/mitsuhiko/minijinja/blob/main/examples/syntax-highlighting/src/main.rs

[esoterra]

Thanks for the example!

I was wondering if call was suited to this, but I really wasn't clear to me how that would work from the Jinja2 docs. I was mostly leaning towards custom block tags because I was basing it on how syntax highlighting support works in Nunjucks for Eleventy.

I'll move forward with the call-based approach, thanks!

[mitsuhiko]

Even back then when I wrote jinja2 I really tried to steer people away from custom tags, but since it was so easy to add them, some utilization kept happening. The unfortunate aspect though is that any custom syntax means custom editor plugins etc. I feel like for as long as you can get away with call that is the way to go.