✨ Add {anywidget} directive and AST node

[stevejpurves]

Upstreaming and adapting the AnyWidget directive code started by Curvenote, to provide a widget directive for MyST with first class AST support.

Note

See jupyter-book/myst-theme#795 for the theme counterpart.

See the tracking issue: #2603

[changeset-bot]

🦋 Changeset detected

Latest commit: e85d070

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 12 packages

Name	Type
myst-frontmatter	Patch
myst-directives	Minor
myst-spec-ext	Patch
myst-common	Patch
myst-cli	Patch
myst-config	Patch
myst-parser	Minor
myst-roles	Minor
myst-to-html	Minor
mystmd	Patch
myst-migrate	Patch
myst-transforms	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[agoose77]

YAAAS! Thanks for sharing this, saves us having to reinvent the wheel!

[choldgraf]

I think that this is a cool enhancement and would definitely relieve some of the customization pressure for users that want to do custom stuff with JS and CSS.

Examples of things I'd want to try documenting (and using) with this:

• Embed custom styling for the handles in the github-handle-links plugin without the hacky abomination I'm using now
• Use a transform to convert a "static" table into a fancy table with datatables

I left a few comments to understand the scope you're aiming for in this PR. That said, I kinda feel like we should try to merge this in quickly so that we can play around with it and iterate in subsequent PRs (e.g. to add more docs, testing, etc once the implementation feels like it has fleshed out).

Should we make a release of mystmd and myst-theme, then merge this in and start iterating from there?

What do you think @stevejpurves ?

[choldgraf]

do we always need to load a JS module? Could somebody use this just to load CSS onto the page or something? My first thought was that this could be a way for a plugin to load custom CSS even if it didn't want to change stuff with JS.

[stevejpurves]

yes, anywidget always loads a JS module. The JS and CSS are rendered within a shadow DOM for isolation, so this is not a vehicle to get global CSS on the page. People can already do that via user style sheets.

[choldgraf]

Got it - so basically styling could be applied only to the DOM elements that are created by the directive.

People can already do that via user style sheets.

To clarify, users can, but plugins cannot.

[rowanc1]

Embed custom styling for the handles in the github-handle-links plugin without the hacky abomination I'm using now

I don't think this PR is aimed at injecting styles in a general sense, but that is something that we should allow plugins to do somehow. This is for sure a hack — taking over the math node. :(

Any widget is mostly aimed at JS (e.g. your second example of the tables) — that could be an any widget + static document to a csv of something, or a map, or a dynamic chart, or basically anything. It skips the need for Jupyter a bit for the web-renderer, which is really nice — but it also can play with Jupyter as well.

Agree on getting this in and iterating, maybe with a beta flag in the docs when it shows up there.

[rowanc1]

Some other small comments.

[stevejpurves]

@choldgraf yes a datatable widget is a nice example, albeit one that might be involved to create unless there is an existing out of the box js library you are looking at wrapping.

note it would be good to get examples in here: https://github.com/jupyter-book/example-widgets as a testing ground as well as more extensive docs

[stevejpurves]

Thanks for all the feedback, I'll do another quick pass on this soon. After that I think some docs is the main blocker for merge - in like with @agoose77's definition of done in the issue.

[agoose77]

In bef3fbf I made some tweaks:

• moved types to myst-spec-ext
• deleted the static files (it's not in use right now)

I also made the options for CSS and ESM path or URL. I'll explain why.

I've realised that artifacts from GH Actions are not CORS-friendly! This is a real pain! It means that people will have to hack around with GitHub Pages if they want to serve content that we can fetch from another page.

Instead, I think we should take this as a signal that local files are important — the same system for loading local ESM is useful in cacheing remote ESM at build time, which ignores CORS.

I have a prototype locally but it's ugly and needs a bit of work. I will push it here to unblock people (42b8ac4), but feel free to delete/redo it as needs be.

Long term I think we should use this as a cue to support path option types in directives etc, so that we can generalise this pattern (or, perhaps better, a utility to "declare" a static asset) in the directive. However, for now it's probably fine to special case widgets.

[agoose77]

Here's a preview of the docs:

@choldgraf you've spent a lot of time in our docs recently. I didn't spend a huge amount of time here because I time-boxed to get the whole PR forward. Could I ask for a glance over?

@stevejpurves can you sign off the changes?

[stevejpurves]

I think this is a good step 👍🏼

[stevejpurves]

should we just be going with widget?

[agoose77]

I'm leaning that way too, actually!

[stevejpurves]

👍🏼

[stevejpurves]

thanks for getting to this @agoose77

[agoose77]

My pleasure!

[stevejpurves]

@agoose77 feedback on changes:

• from string -> url/file: I think the file copying change is definitely what we want, perhaps there should be an option which is an escape hatch to allow it to remain remote only, when remote? This implementation is good enough for this cut (but also if generalised could solve the other "get static files into the _build folder" issues/requests).
• just related to that should we support inline esm/css like this? or the local files are good enough/better? I feel now like there is no need? question for next PR though!

[stevejpurves]

I am testing and seeing issues when using the jupyter-book/example-widgets repo;

I am using the latest from this branch jupyter-book/myst-theme#795

@agoose77 did you get the examples repo to work as expected after your change?

[agoose77]

@stevejpurves and I discussed this via DM on Discord — I forgot an await, which introduced a race race condition.

from string -> url/file: I think the file copying change is definitely what we want, perhaps there should be an option which is an escape hatch to allow it to remain remote only, when remote? This implementation is good enough for this cut (but also if generalised could solve the other "get static files into the _build folder" issues/requests).

Yes, I definitely think this should be generalised. I want to tackle that separately, so that it doesn't block this work. See more here: #2714!

just related to that should we support inline esm/css like this? or the local files are good enough/better? I feel now like there is no need? question for next PR though!

I think ESM and CSS in-document is an antipattern for us. I'm also thinking about whether the same is true for JSON. For simple widgets, inline JSON is probably preferable, but for bigger ones we don't want huge 3D model definitions in the bundle! We can probably add a separate model-file directive argument for that and combine it with a transform that moves JSON to disk. I think it's OK to hold off on that and introduce backwards compatibility down the road?

[stevejpurves]

Yes, I definitely think this should be generalised. I want to tackle that separately, so that it doesn't block this work. See more here: #2714!

💯 on that @agoose77 and thanks for the replies on DM

[choldgraf]

I took a pass at the documentation to build on what @agoose77 has added (as a compliment to the docs work I added over in the myst-theme PR). I think overall this is good enough to ship as-is, especially now that we have local modules, which IMO makes this much more powerful because you can wrap other libraries in your own anywidget logic).

When I did that, I also updated the nomenclature to use the phrase MyST Widgets and then Widgets (since it seemed like both @agoose77 and @stevejpurves were in favor, and I also think this is a good idea). I have not changed any filenames or code, because I don't want to clobber somebody's work if y'all have working branches locally (I assumed the docs were safer to change).

IMO, this is good enough to ship right now, if we do:

• label it as an alpha feature with an admonition at the top (ie, no promises that the API or key functionality will be stable, but we quickly hope to converge on beta-level stability)
• (optionally) create an issue where people can provide comments etc, but IMO we could also just ask people to open issues if they hit problems

That way we can invite experimentation from power users without setting up the assumption of stability.

I think this is a really cool feature, and leans into our strategy of making myst a more hackable/pluggable platform for developers!

[stevejpurves]

I think we can merge as soon as CI is green

[stevejpurves]

I am approving this based on out of band convos with @agoose77