🪐 Allow for Jupyter execution (including inline) directly in mystmd CLI

[agoose77]

This PR adds execution support to the mystmd CLI. A sketch at some docs below from @rowanc1!

• add health testing to Jupyter Server selection logic
• prewarm kernels?

[rowanc1]

Figured out my problem, there was a dead server hanging about that could not be removed. I went into the runtime using jupyter --paths and deleted it manually.

As part of that testing I went through and changed a few things:

• improved some of the logging
• For errors: I made things that were happening (jupyter starting) be in log and things that were node/execution errors part of the fileError. The logs trigger immediately, while the ones in the vfile don't get logged for a while and are stored. I think this is a decent pattern, and a bit of a nuance on the advice I gave earlier.
• You now need to pass the --execute flag from the CLI, the default is false.
• You can clean the execution cache using myst clean --execute

Things I noticed:

• The jupyter server stays around and isn't shut down when running myst build --execute. Or myst build --execute --html this means that the process doesn't stop and just hangs. Need to investigate how to kill the server.
• I put the execute flag in, but this should maybe also be in the myst.yml?
• I don't think we are passing the minification flags around anymore? This needs to come back.

[agoose77]

There's something wacky going on with Node - it refuses to quit even when the child process has been killed. There are tools to ask node why it's not quitting, but I don't have time right now to dig into it. It might be that we want to remove the use of closures and move this logic into a class. That might be less opaque to node, if refcycles are the problem.

[agoose77]

@fwkoch I changed the Python package from using setuptools to using the Hatch build backend. The benefits of this are fewer config files, more declarative config, and the ability to use plugins to pull metadata from mystmd's package.json. Are you comfortable with this change?

I can split it into a new PR if we need.

[rowanc1]

Oh cool, this is nice.

[agoose77]

To be clear here, the version is only updated at pypi deployment time (because it's taken from package.json, which is copied in the deploy bash scipt).

[rowanc1]

I am still a bit concerned that we are no longer passing this option in at all? I think this actually might be breaking JATS - @fwkoch I would love another set of eyes here.

[agoose77]

@rowanc1 I've updated the cache key struct to include the raises-exception tag that allows notebooks to ignore errors. I haven't included a cache-version in the key; my rationale is that this is implicitly part of the hash - if we change the structure, the hash will change.

[fwkoch]

Ok - putting my approval on this. PR looks great! Lots of tedium passing the execute flag around - maybe those CLI options could be a refactor target some time.

Regarding the minifyMaxCharacter stuff - I played around with static outputs and I think it's all fine...? During mdast processing we call transformOutputsToCache - this does the exact same minification we were also doing during notebook loading. So now (since minification was removed from loadFile) it just always happens in that transform instead of both places. I still have some vague worries that there was an edge case causing us to do this in two places, but no test coverage and I couldn't recreate it 🤷‍♀️ Good to land, and we can just keep an eye on it.

[rowanc1]

Thanks @fwkoch for your review -- and @agoose77 for going back and forth on this. I am hugely excited to get this out and start testing it in the wild. I have been using it for the last several days and it is working well!

🚀

[rowanc1]

Ah -- we didn't add changesets @agoose77! :)