{% markdown %} ... {% endmarkdown %} template tag

[simonw]

While using render_template() for pages in https://datasette.io/tutorials I kept running into this annoyance:

{{{ render_markdown("""
# Data analysis with SQLite and Python
"example" and 'example'
""") }}}

This raises an error - the double quotes need to be \" escaped.

If you switch to ''' then you need to use \' for the single quotes instead.

After some exploration, it looks like the solution is to implement this:

{% markdown %}
# Data analysis with SQLite and Python
"example" and 'example'

{% endmarkdown %}

[simonw]

It's a bit hard finding examples of custom tags like this. The best I found was actually another Markdown one:

https://github.com/jpsca/jinja-markdown/blob/main/jinja_markdown/__init__.py

Markdown is also used as the example in this tutorial:

https://ron.sh/how-to-write-a-jinja2-extension/

https://github.com/jinrudals/jinja2_script/blob/main/jinja_script_block/__init__.py is good too.

[simonw]

Neither of those Markdown ones are quite what I want though - I would like to be able to pass custom arguments to it, as seen here:

{{ render_markdown("""
## Markdown table

First Header  | Second Header
------------- | -------------
Content Cell  | Content Cell
Content Cell  | Content Cell
""", extensions=["tables"],
    extra_tags=["table", "thead", "tr", "th", "td", "tbody"])) }}

I want it to work like this:

{% markdown extensions='["tables"]' extra_tags='["table", "thead", "tr", "th", "td", "tbody"]' %}
## Markdown table

First Header  | Second Header
------------- | -------------
Content Cell  | Content Cell
Content Cell  | Content Cell
{% endmarkdown %}

I'm using JSON objects in single quotes here, I think that's probably the easiest option. Or I could do this:

{% markdown extensions="tables" extra_tags="table thead tr th td tbody" %}

And then split on spaces. But that's harder for the other option, extra_attrs:

extra_attrs={"a": ["name", "href"]}

I could keep that as JSON, or I could invite syntax for it:

{% markdown extra_attrs="a:name,href span:id,class" %}

I quite like that invented syntax, actually.

[simonw]

Parsing attributes in custom Markdown tags turns out to be pretty hard! I figured this out, eventually:

    def parse(self, parser):
        lineno = next(parser.stream).lineno

        # Gather tokens up to the next block_end ('%}')
        gathered = []
        while parser.stream.current.type != "block_end":
            gathered.append(next(parser.stream))

        # If all has gone well, we will have a sequence of triples of tokens:
        #   (type='name, value='attribute name'),
        #   (type='assign', value='='),
        #   (type='string', value='attribute value')
        # Anything else is a parse error

        if len(gathered) % 3 != 0:
            raise TemplateSyntaxError("Invalid syntax for markdown tag", lineno)
        attrs = {}
        for i in range(0, len(gathered), 3):
            if (
                gathered[i].type != "name"
                or gathered[i + 1].type != "assign"
                or gathered[i + 2].type != "string"
            ):
                raise TemplateSyntaxError(
                    (
                        "Invalid syntax for markdown attribute - got "
                        "'{}', should be name=\"value\"".format(
                            "".join([str(t.value) for t in gathered[i : i + 3]]),
                        )
                    ),
                    lineno,
                )
            attrs[gathered[i].value] = gathered[i + 2].value

        body = parser.parse_statements(["name:endmarkdown"], drop_needle=True)

        return nodes.CallBlock(
            self.call_method("_render_markdown", []), [], [], body
        ).set_lineno(lineno)

[simonw]

I used ChatGPT to help get me to this point. The code it wrote was almost entirely broken, but it gave me enough hints that I could figure out how to build a non-broken version https://chat.openai.com/share/ff041b92-992d-40f0-b411-42a49ff0b3e3

[simonw]

I got the JSON attributes version working. I'm going to check that in with tests, then maybe do the non-JSON syntax alternative.

[simonw]

datasette-render-markdown/datasette_render_markdown/__init__.py

Lines 101 to 158 in 3055c91

	class MarkdownExtension(Extension):
	tags = set(["markdown"])
	def __init__(self, environment):
	super(MarkdownExtension, self).__init__(environment)
	def parse(self, parser):
	# We need this for reporting errors
	lineno = next(parser.stream).lineno
	# Gather tokens up to the next block_end ('%}')
	gathered = []
	while parser.stream.current.type != "block_end":
	gathered.append(next(parser.stream))
	# If all has gone well, we will have a sequence of triples of tokens:
	# (type='name, value='attribute name'),
	# (type='assign', value='='),
	# (type='string', value='attribute value')
	# Anything else is a parse error
	if len(gathered) % 3 != 0:
	raise TemplateSyntaxError("Invalid syntax for markdown tag", lineno)
	attrs = {}
	for i in range(0, len(gathered), 3):
	if (
	gathered[i].type != "name"
	or gathered[i + 1].type != "assign"
	or gathered[i + 2].type != "string"
	):
	raise TemplateSyntaxError(
	(
	"Invalid syntax for markdown attribute - got "
	"'{}', should be name=\"value\"".format(
	"".join([str(t.value) for t in gathered[i : i + 3]]),
	)
	),
	lineno,
	)
	attrs[gathered[i].value] = json.loads(gathered[i + 2].value)
	# Validate the attributes
	# raise TemplateSyntaxError("attrs: {}".format(attrs), lineno)
	body = parser.parse_statements(["name:endmarkdown"], drop_needle=True)
	return nodes.CallBlock(
	# I couldn't figure out how to send attrs to the _render_markdown
	# method other than json.dumps and then passing as a nodes.Const
	self.call_method("_render_markdown", [nodes.Const(json.dumps(attrs))]),
	[],
	[],
	body,
	).set_lineno(lineno)
	async def _render_markdown(self, attrs_json, caller):
	attrs = json.loads(attrs_json)
	return render_markdown(await caller(), **attrs)

[simonw]

I had to do async def _render_markdown(self, attrs_json, caller): for this to work - I wonder if that would break in Jinja run outside of async mode?

[simonw]

Just needs documentation now and I can release it.

[simonw]

Documentation: https://github.com/simonw/datasette-render-markdown/blob/bfa53f6621e52004d3c16cc5b8ae8815ba5c5d07/README.md#markdown-in-templates

[simonw]

Wrote a TIL about this: https://til.simonwillison.net/jinja/custom-jinja-tags-with-attributes