Loops?

[abirmingham]

Hello!

I'm evaluating evaluating remark for a project where I need to render templates to markdown. Currently I am using jinja, but this means that I have two passes - python and javascript - and there is brittleness around encoding issues. Finally, adding the remark ecosystem to my project means that I can take advantage of various remark features.

However, one thing I am noticing about remark is that there is not a builtin or plugin capability for loops. In my jinja templates, I commonly have something like the following:

# Items
    {% for item in list %}
        * `{% item %}`
    {% endfor %}

In remark, it appears I can write a custom component (or perhaps a directive?) to achieve this functionality generically. However, the lack of an existing plugin makes me wonder if I'm thinking about remark incorrectly, and/or trying to make it do something that it is not intended to do. Is this the case? If not, what is the best way to achieve the above example?

Thanks!

EDIT: note that there is a bug in the above example that I did not immediately notice, which is that item might contain markdown, and I am not escaping that markdown. This is one example of how the "two pass" approach of jinja+markdown is brittle (though, to be fair, jinja does support an autoescape function).

[ChristianMurphy]

CommonMark and GitHub Flavor Markdown (GFM) are both document languages, they have neither a concept nor an implementation of anything Jinja-template-like.
Their goal is simplicity for static content, templates are not particularly simple or static 😅

To give a better answer there are a few pieces of context which would be needed:

• Who is authoring these documents? Are they writers, developers, something else?
• What kinds of content are you trying to generate? (And what brittleness have you experienced)
• What is the source of the values being passed into the templates?

Without those answers I can only offer some sweeping generalizations of possible solutions, which may or may not work for your specific use case.

1. Create a plugin to edit the Abstract Syntax Tree

Working directly with text can inherently be a bit brittle, that is what remark provides an Abstract Syntax Tree (AST) called mdast https://github.com/syntax-tree/mdast
For example a header could be used, and content can be automatically injected below that, see https://github.com/remarkjs/remark-toc for on example of how this can work.
The Unified docs https://unifiedjs.com/learn/ provide a bit more context on working with plugins and ASTs.

2. Use remark-directive and create a plugin to replace directives

remark-directive is an option, it is a semi-standard way of declaring new block types, and the block/directive attributes could point to the data source it should pull from.
If the data source were a JSON file, an example markdown file could look like:

# items

:::list-from-json{file-path='./list.json'}

3. Use MDX and use JS or Components to add content

MDX adds JSX to Markdown https://mdxjs.com/
JSX is a flavor of JavaScript, with shorthand for adding html and components, it also can embed JavaScript, which does have loops.

An example of a MDX file with a loop

# item

# items

<ul>
  {props.items.map((item) => (
    <li>{item}</li>
  ))}
</ul>

An example of a MDX file using a component to generate content based off a JSON file:

# items

<ListFromJson file="./items.json" />

[abirmingham]

Christian -

Firstly, thank you so much for your incredibly thoughtful and comprehensive answer. I appreciate you being so generous with your time.

Secondly, here are the answers to your questions:

Who is authoring these documents? Are they writers, developers, something else?

The template authors fall into two groups: A) technical writers who are not familiar with javascript, and B) researchers who regularly write javascript. Note that human-readable strings could in theory be separated out - and perhaps they should for internationalization purposes - and in that case only folks in group B would be dealing with these documents.

What kinds of content are you trying to generate?

The product detects anomalous or potentially malicious events seen on the network and generates "detections" to describe these events to the user. The descriptions are currently jinja with markdown. Jinja is used both for control directives (loops, if statements, etc) and for variable interpolation. More on the variable interpolation later...

(And what brittleness have you experienced)

The primary issue we've encountered is with interpolating strings which may contain backticks. The problem is that these variables need to be interpolated differently depending on surrounding context - specifically whether the context is already a code span. Outside a code span, the backticks can simply be escaped. Inside a "double backtick code span", the backticks should not be escaped. Inside a "single backtick code span", I am unaware of a solution outside of removing the backticks from the variable (or modifying the surrounding context to convert to use double backticks). You can see here why a single pass render to react is desirable :)

Secondly, we have in-browser code that scans list items that appear to be URLs, and we attempt to pattern match on these URLs in order to present them in a special way as react components. These have broken in the past when the list item structure changed. Additionally, there is no obvious way to pass arguments to these "components". This points us towards a remark directive, mdx, Markdoc, etc.

Finally, template authors do not presently have a way of previewing templates, which means they can easily write a jinja template that produces broken markdown. If we eliminate jinja in favor of a pure-js solution, we are able to more easily provide a sandbox environment for authors. And, just as importantly, we could use portions of that code to programmatically validate the templates in merge requests. Jinja does not eliminate preview/validation but it does make them more challenging.

What is the source of the values being passed into the templates?

Some of the values are defined by us and are known to be safe. Some of the values are received from third parties. This means that in general we cannot trust the values. Further, we cannot trust template authors (and MR readers) to be sufficiently diligent to run all interpolated values through an escape function (though it's possible that we could write a linter that scans for variable interpolation, but it seems like an AST would be a better target for linting). Jinja offers a relatively low risk assessment (an autoescape function can be globally defined), and the attack surface is probably limited to react-markdown exploits. Something like MDX is probably higher risk, though I am not yet well-versed enough to understand attack surface there.

On the other hand, jinja is appealing because the templates are already written in jinja, and we do not currently have a javascript service on the server (which would be needed for a use case of offline rendering).

Any advice/guidance you could offer would be greatly appreciated :)

[ChristianMurphy]

To make sure I understand, some of the keypoint's I'm hearing:

• Incoming content is not (completely) trusted
• Content authors are partially non-technical, but all are technical enough to write Python/Jinja?
• Content also needs to be internationalized, and that is part of the markdown?

I'm not sure any solution I'm aware of in the Python or JS ecosystem can check all those boxes 😅
remark-directives sounds like it may be a good fit for the first two. It does not solve internationalization though.

The primary issue we've encountered is with interpolating strings which may contain backticks. The problem is that these variables need to be interpolated differently depending on surrounding context - specifically whether the context is already a code span.
...
You can see here why a single pass render to react is desirable

Makes complete sense, and specifically I think that an Abstract Syntax Tree (AST) would fit well here.
It has the context of what's inside of it and surrounding it, and can adapt accordingly.
remark and rehype have a plugin system which allow working with the AST, documented more here: https://unifiedjs.com/learn/

On the other hand, jinja is appealing because the templates are already written in jinja, and we do not currently have a javascript service on the server (which would be needed for a use case of offline rendering).

Completely understand.
I don't think there will be a pure Python port of remark anytime soon.
But, it could be possible to write Python bindings for https://github.com/wooorm/markdown-rs, the new Rust implementation of micromark and remark.
Those bindings could be written using something like https://github.com/PyO3/pyo3, and exposed to Python.
Which would open the door to tighter integration with Jinja and other Python based templating systems.

Or, alternatively and as you note, server side JavaScript could indeed be added to do server side rendering.