"Q&A against documentation" on the datasette.io homepage

[simonw]

[simonw]

Here's the YAML:

https://github.com/simonw/datasette.io/blob/b821fb19eda08b4942183db507bb6f986f8134bf/news.yaml#L3

The markdown is stored in the DB:

https://datasette.io/content/news?_sort=rowid&date__exact=2023-01-13

And rendered here:

https://github.com/simonw/datasette.io/blob/b821fb19eda08b4942183db507bb6f986f8134bf/templates/index.html#L112

https://github.com/simonw/datasette.io/blob/b821fb19eda08b4942183db507bb6f986f8134bf/templates/pages/news.html#L38

[simonw]

So it looks like datasette-render-markdown is the thing that renders & as & in this context.

It uses https://python-markdown.github.io/

[simonw]

This tool compares different markdown implementations:

https://babelmark.github.io/?text=%5BSemantic+search+answers%3A+Q%26A+against+documentation+with+GPT3+%2B+OpenAI+embeddings%5D(https%3A%2F%2Fsimonwillison.net%2F2023%2FJan%2F13%2Fsemantic-search-answers%2F)+shows+how+Datasette+can+be+used+to+implement+semantic+search+and+build+a+system+for+answering+questions+against+an+existing+corpus+of+text%2C+using+two+new+plugins%3A+%5Bdatasette-openai%5D(https%3A%2F%2Fdatasette.io%2Fplugins%2Fdatasette-openai)+and+%5Bdatasette-faiss%5D(https%3A%2F%2Fdatasette.io%2Fplugins%2Fdatasette-faiss)%2C+and+a+new+tool%3A+%5Bopenai-to-sqlite%5D(https%3A%2F%2Fdatasette.io%2Ftools%2Fopenai-to-sqlite).+

It suggests that python-markdown renders this just fine:

[simonw]

I think this is likely a bug in the interaction between the markdown rendering and Bleach in this plugin:

datasette-render-markdown/datasette_render_markdown/__init__.py

Lines 75 to 80 in c04b0b6

	html = bleach.linkify(
	cleaner.clean(
	markdown.markdown(value, output_format="html5", extensions=extensions or [])
	),
	skip_tags=["pre"],
	)

I can recreate it locally like this:

>>> from datasette_render_markdown import render_markdown
>>> render_markdown('[this & that](https://www.example.com/)')
Markup('<div style="white-space: normal"><p><a href="https://www.example.com/" rel="nofollow">this &amp;amp; that</a></p></div>')

Note this & that in the output.

[simonw]

Confirmed: I removed the calls to bleach and got this:

<div style="white-space: normal"><p><a href="https://www.example.com/">this &amp; that</a></p></div>

[simonw]

After more exploration, it turns out it's the call to bleach.linkify(...) that causes the double escaping of the ampersand.

[simonw]

https://bleach.readthedocs.io/en/latest/linkify.html says:

If you plan to sanitize/clean the text and linkify it, you should do that in a single pass using LinkifyFilter. This is faster and it'll use the list of allowed tags from clean.

[simonw]

Deployed that fix to https://datasette.io/