Develop extensions inside JupyterLab itself

[saulshanabrook]

It would be awesome if we could develop extensions inside of JupyterLab itself. A robust extension development community is essential for JupyterLab to flourish and making it easier to get started might help with this.

The user story would be like:

1. Run "Create new extension" command
2. A new text file opens called extension.tsx that has a tempalte for an extension.
3. It is automatically loaded in JupyterLab without reloads
4. As you edit this file, the extension is reloaded.

This would help a lot when getting people up and running to develop extensions, say in a tutorial setting.

There would then need to be some additional steps to set them up to distribute their extension on NPM with a proper build system.

Required technical steps

• Using @krassowski's work on a LSP server for JupyerLab so we can TypeScript hints: https://github.com/krassowski/jupyterlab-lsp
• Add ability to load modules at runtime in JupyterLab, so we can resolve import statements client side or create server service to compile JS files.

[telamonian]

This kind of dogfooding is a common IDE feature, and I agree it would be nice. I think for the live reload you need some kind of Hot Module Replacement. From the little I've gleaned, there is some way to get webpack to do this. I do not have a solid grasp of how hard this is to set up, but my sense is that it's pretty different from our current build set up, so 🤷‍♂. It would be pretty cool.

Does anyone know anyone from Webpack, or one of the up and coming alternative bundlers? We could really use some domain specific expertise as we try and work out the next version of the build and extension systems.

Instead of a .tsx script, do you think we could make this a notebook story (that is supposed to be our strong suit)? Set up appropriately, it could be a nice stepping stone to work your way up to and/or prototype full extension packages (just like notebooks are for every other application). There is at least one Typescript Jupyter kernel. Of course it can't talk directly to the code running in a user's client, but maybe there's a way around that? At least for a certain well-defined set of entry points that we happen to have lying around.

It would be dang cool to be able to import random tokens into a cell and be able to mess around with their corresponding instances. Feasible?

[bollwyvl]

Did somebody say "kernels" in the browser with access to lab itself? I haven't updated jyve for 1.x since a lot of things changed in the kernel/session stuff, but would be more than willing to warm it back up, fight for the extension points it would need, contribute it to core, whatever. I think combined with the jupyterlab-lsp features it could be viable indeed. Who even wants to write typescript? How about writing labextensions in python in lab with pyodide? I jest, but... really, anything we can do to help people get on-board is worth it.

webpack-dev-server: while it's a mere additional 54 dev dependencies, you then start adding more stuff... and i'm loathe to make every end user install them. as you mention, webpack is not the endgame.

I wish there was more time to invest in making the existing build work better (runtime chunks, service workers, etc) in ways that would benefit every user, but the whole js tooling thing is exhausting, and on most of my projects, I build a lab in ci and ship without nodejs in the lab env because 200 mb of node_modules, half as much in ~/.cache/yarn, and windows, and memory, node-gyp, and too many open files, and... the long game can really only be moving the whole as-distributed shooting match to ES modules that are put in place and just work for end users... 100s of requests be damned, we add a serviceworker cache... but i think a few little things like, well, react, aren't really on board yet. Sigh.

Notebooks as extensions is a whacking good idea. metadata/labextension/package.json, maybe a settings/schema (with a (meta)schema-constrained editor for them... and no new deps), first markdown cell is README.md, and off you go. Heck, you could probably write a cookiecutter/yeoman task today to make that build a real package, but the magic would be in hacking it live, being able to send it to a friend, have them drop it someplace, and try out your new freaky lab, remote code execution and all.

[krassowski]

@saulshanabrook is hot replacement/autoreload on the server-side (see @telamonian's jupyter/notebook#4795) a part of this vision as well or is it limited to frontend side?

[telamonian]

@krassowski That is a good question. I think that the focus for this issue should be on labextensions, not for conceptual reasons but for practical ones. Autoreload for server extensions involves a completely different part of the Jupyter stack, and a different set of problems.

Basically, the autoreload in jupyter/notebook#4795 is an improvement over manually restarting the Notebook server, but only a minimal one. Currently (given the default browser setup), when Notebook autoreload is enabled, the browser/jupyterlab client will restart along with the server. That can probably be worked around (either via (tornado.autoreload.add_reload_hook)[https://www.tornadoweb.org/en/stable/autoreload.html#tornado.autoreload.add_reload_hook] or by just moving the browser handling outside of any tornado app), but it remains to be done.

[telamonian]

Also, since the server takes a good minute to start up in (typescript) watch mode, these two forms of autoreload will clash with each other, somewhat. A possible fix would be to better separate the part of the server that handles build and the part of the server that does the api route handling (the latter of which is probably what you'd want to autoreload when developing server extensions). Maybe this can all be wrapped up with a bow as part of the migration away from Notebook (ie #7416)?

[krassowski]

For some time now https://github.com/jupyterlab/jupyterlab-plugin-playground enables developing extensions inside JupyterLab itself and https://github.com/jupyterlab/ui-profiler enables profiling performance of the UI inside JupyterLab itself (in future will also enable profiling of extensions, jupyterlab/ui-profiler#18).

Not closing this issue as this still needs to be documented.

[erkin98]

any updates on hot/live reloading?