Picture-generating blocks

[pveber]

Following the discussion on discord, I'm opening a discussion to see if mdx could be extended to support graphic outputs. I have prototyped a tool that reads a markdown file with ocaml blocks and produces an html file with toplevel outputs + rendered pictures in the document. My goal here is to see if I can help providing a similar functionality in mdx.

I have to admit it's not totally clear to me how this feature would fit with how mdx works currently. First question is, would that feature make sense in the pp and test modes? Or would it be better to include it only in a new mode (like the jupyter notebook output, or a new mode for markdown to markdown transformation)?

Also, what would the API for picture rendering? How should pictures be rendered in the final document? In my prototype I started using the install_printer directive of the toplevel to install a printer for a dedicated image type. The printer would render values of this type as HTML img elements with embedded data using an src attribute of the form src="data:image/png;base64 ...". Registering a printer is however not very appropriate, notably because the image values are also rendered as HTML when they appear within a list for instance, and this is then pretty useless and inconvenient. I implemented another mechanism where one explicitly calls a show function to have a picture rendered in the document, and the picture is interleaved with toplevel output. The implementation is less pretty than the output though, as it relies on side effects and a call to Obj.magic... There are certainly better ways to proceed.

Let me know your opinion, and tell me if I can help in some way or another.

[XVilka]

Please also keep in mind animated images like GIF (though it's better to let it go in 2020 already!), WebP, AVIF or WebM blocks.

[yminsky]

One idea I've had about this is around graphs and other visualizations. You could imagine capturing the raw data backing a graph in the text of the mdx (which can then get committed to your source repo), and then being able to convert this into a proper visualization, either in a properly equipped editor (like vscode?) or in web-output. That way, diffs that affected the data could be meaningfully viewed with an ordinary textual diff program, which could be quite useful for various kinds of number-crunching applications.

[pveber]

An HTML element (img or table for instance) could definitely serve that purpose, with the benefit that the syntax is already somehow part of markdown (and already understood by tools like pandoc).

[gpetiot]

I'm not sure it is in the scope of mdx, as mdx focuses on executable blocks, and not on rendering the graphic contents of markdown files, is it not possible to pipe mdx with another rendering tool?

[pveber]

😲
I'm sad I've not even once thought of that. So basically, you call a function that prints an encoding of a picture, and that part of the toplevel output is detected and transformed by an independent tool. I see no downside, and it is so much simpler (and composable) than what I've been prototyping. I'm closing the issue, thanks a lot!

[NathanReb]

I think @gpetiot is right, it feels like this is out of the scope of MDX. What we could eventually consider to make it easier to implement such an independent tool is to extract bits of MDX parsing logic into a small library. We already considered this for formatting related issues.