JupyterLab 3 support?

[bollwyvl]

Hello, and thanks for this marvelous tool.

I'd very much like to see this up and running in JupyterLab 3... it constitutes something of a hard break, as the primary distribution would be via only pip install (though nice devs also continue to publish to npm, so that others can extend their stuff). The benefits for users are very significant: no more runtime nodejs, mysterious downloads of 100s of MiB from npmjs.org, a single source of version truth per package, etc. The development workflow is a little weird, more like the nbextension, as it relies on data_files to put stuff in a place on disk, but being able to build once on linux, and test built wheels/sdist everywhere else is priceless.

Happy to help in any way I can!

Blocked By

• pythreejs
  • feat: jupyter lab 3.0 federated bundle support jupyter-widgets/pythreejs#342

[bernhard-42]

@bollwyvl Yeah, I agree, Jupyterlab 3 solves a lot of the deployment issues. And it is definitely top of my list.
However, pythreejs is currently still not supported (jupyter-widgets/pythreejs#345). As you can see there, I already asked for an ETA ...
Additionally, I had some real bad experience when I migrated some other tool from Jupyterlab 1.x to 2.0 - which kind of let's me think that Jupyterlab 3.1 is the one to go for - if pythreejs is supported by then.

[bollwyvl]

Great, I'll update the above with the punch list (and stop preaching to the choir), and see if there are upstreams I can help on.

I don't anticipate 3.1 being a major burden for "pure" widget packages, and the delta between code in 2/3 (unlike the lumino thing, and the icon disaster) is not that big, just the packaging business.

[bollwyvl]

Pushing to get the CI up and running for pythreejs, which should help review, etc.

[bollwyvl]

It's a little hard to tell what this repo actually depends on (vs demo stuff) so if you know which extensions are required, please add them, and I'll see what i can do...

[bernhard-42]

I don't anticipate 3.1 being a major burden for "pure" widget packages, and the delta between code in 2/3 (unlike the lumino thing, and the icon disaster) is not that big, just the packaging business.

By waiting for 3.1 I just meant to wait until the incompatibilities with Jupyterlab 3 and some other stuff are gone. Just being on the very careful/coward side here. With Jupyterlab 2 I spent days to find the right combination of python module versions and npmjs versions for my two major jupyter based projects.

It's a little hard to tell what this repo actually depends on (vs demo stuff) so if you know which extensions are required, please add them, and I'll see what i can do...

jupyterlab-cadquery needs what is shown in labextensions.txt:

• @jupyter-widgets/jupyterlab-manager@2.0.0
• @jupyter-widgets/jupyterlab-sidecar@0.5.0
• jupyterlab-datawidgets@6.3.0
• jupyter-threejs@v2.2.0
• jupyter_cadquery@v1.0.0

@jupyter-widgets/jupyterlab-sidecarseems to be already "ported" to JL3, see https://github.com/jupyter-widgets/jupyterlab-sidecar

jupyterlab-datawidgets came with some module, no idea, but it seemed to be necessary in the past. Maybe we don't need it any more in JL3? Worth a try because it might not be ported by now.

The jupyter_cadquery stuff I need to repackage.

[bernhard-42]

Just tested it, jupyterlab-datawidgets automatically comes with conda when you conda env create -f ./environment.yml -n cq2-jl. Currently no idea who requires it. I actually don't use it in my tool.

[bollwyvl]

no idea who requires it ... don't need it any more in JL3

Yeah, Datawidgets is a dep of jupyter-threejs, and it's how those sweet, sweet binay buffers get to the browser without becoming strings. It's ready to go, and will automagically get pulled in.

days to find the right combination of python module versions and npmjs versions

yeah, the whole node vs python thing (or really, node at all, for users) was not fun, especially on windows. Hence my interest in getting out of those particular woods, and back to building cool stuff. on the js side, typescript is the best tool in the shed for keeping stuff sane, but that's a whole other ball of fish.

waiting for 3.1

Sure, that's great... the only thing is sometimes extensions that are n degrees removed uncover weirdness

Three.js was actually one of the motivators in us starting with webpack, as require.js/umd was not up to the task of handling multiple THREEs in the page. The federated stuff required getting a bunch of stuff into webpack itself, and is some serious dark magic. About the only thing you'll have to worry about is a few more lines in package.json#/jupyterlab around shared packages, but from the links on this issue, it should be pretty clear what you need (probably @jupyter-widgets/base, which is not a labextension, but won't tolerate having multiple copies, even of the same version on the page).

The remaining packaging weirdness, setuptools data_files, will hopefully become optional, and then deprecated, but it's a slow ship to turn.

[tescalada]

The PR this issue is blocked by was closed and replaced with jupyter-widgets/pythreejs#350, which is now merged. I'm not totally clear on if that means this is unblocked or not, but I figured I would mention it while I was looking around to see if there was anything I could help with. I'll try to setup my environment so I can actually help with testing/building/whatever.

[bollwyvl]

Right: merged, but no release yet. I've been trying to get the CI up and running, as it seems like not having viable logs is hurting progress.

[bernhard-42]

@bollwyvl @tescalada I have successfully split my code and moved the widget stuff into a separate repository (https://github.com/bernhard-42/jupyter-cadquery-widgets) and deployed jupyter-cadquery-widgets to pypi. jupyter-cadquery now uses JupyterLab 3 and installs the the widget code from pypi.

While it was still a hassle to migrate from JL 2 to 3 (unravelling my code, and then dealing with the quirks of typescript in a javascript project) the result is amazing. The installation now is a simple pip install => NICE!!!

ipywidgets, pythreejs and sidecar worked flawlessly and I've just released 2.0.0.rc1 of jupyter-cadquery to pypi so there is one JL2 project less in the world :-)

Thanks for your help, really appreciated.

[bollwyvl]

Very, very awesome. Trying the binder out now!

• looks great1
• html output at the end of 1-cadquery doesn't load (lots of thus-and-such not found, presumaly pythreejs stuff).

Once you cut a for-real 2.0.0 release, i'll fire up the conda-forge machinery... let me know if you would like to co-maintain, but the burden is much lower for downstreams with lab3.

[bernhard-42]

@bollwyvl What do you mean with

html output at the end of 1-cadquery doesn't load (lots of thus-and-such not found, presumaly pythreejs stuff).

I've just just tested it in Chrome and everything works as expected. If it still shows errors for you, maybe you could add a screeenshot?

Btw. if you look at assemblies/2-hexapod.ipynb in binder you can see pythreejs animation system in action. I was surprised how smooth that works even on binder.

I will iron out some rough edges around binder and work on documentation. If done I will release 2.0.0.

What do you mean by "conda-forge machinery" - never looked into it ...
Would that mean jupyter-cadquery would then be available via conda install -c conda-forge ?
And of course, happy to co-maintain.

[bollwyvl]

how smooth that works even on binder.

yeah, that's the beauty of Other Peoples' Computers, and passing geometry over the binary pipe 🎉 !

conda-forge machinery

right. for my purposes, if it isn't on conda-forge, i can't use it 😊 .

in a local clone of https://github.com/conda-forge/staged-recipes i'd run:

cd recipes
grayskull pypi jupyter-cadquery

that would generate a meta.yaml, and know how to map install_requires, python_requires, find the license, etc.

more importantly, it will show packages that aren't available yet... so probably cadquery 2.1.

you run it once locally, if it works, you push it and make a PR, it builds on lin/osx/win, and somebody reviews it. they either kick it back and we fix it, or they Press The Button, and within the hour (usually) it's up on conda-forge.

Then, every time the upstream (you in this case) makes a new pypi release, a bot makes a PR to the feedstock (can take a couple hours), and after it passes, we Press The Button, and within the hour it's up on conda-forge.

If we feel really good about it (e.g. it runs all the tests, with coverage, and a threshold) we can turn on automerge, and it will just happen... but i'm always really scared of that, even though i maintain x00 packages (i try not to look)... including the cadquery. So i probably gotta go check out what needs doing. I'll also take a quick look at your rc tarball and see if it "smells good"...

[bollwyvl]

ok, made #36 and #37, otherwise lookin good!

[bernhard-42]

Cool, as soon as I have 2.0.0 I'll try the conda-forge steps.

Do I need the conda forge steps also for juypter-cadquery-widgets or is thew main project sufficient?

[bollwyvl]

I've just just tested it in Chrome and everything works as expected. If it still shows errors for you, maybe you could add a screeenshot?

• open 1-cadquery.ipynb
• press ⏩
• right click export.html -> Open in new browser tab

[tescalada]

Trying to test out the new build, 2.0.0-rc1 is not yet pushed/tagged in dockerhub.

[bollwyvl]

conda forge steps also for juypter-cadquery-widgets or is thew main project sufficient?

it would run the thing, and throw up some shouty red text about how the dependency is missing. one can submit multiple recipes in a single feedstock.

only do this if it's useful and exciting for you, otherwise i have no problem getting it going!

[bernhard-42]

@tescalada My upload of the docker image is still running :-(
slow upload today

[bernhard-42]

@bollwyvl

open 1-cadquery.ipynb
press ⏩
right click export.html -> Open in new browser tab

The export uses the ipywidgets embedding feature. Not really useful for binder, was more for local documentation of your model. Will remove it from the example and explain in docs, since it seems to be confusing.
Thanks for testing and the feedback

[bollwyvl]

Having a look down the chain... there are layers and layers of build stuff to get cadquery 2.1, so i wouldn't worry too much about it today...

[bernhard-42]

@tescalada Finally, the docker image is uploaded to docker hub and docker run -it --rm -p 8888:8888 bwalter42/jupyter_cadquery:2.0.0-rc1 works as expected (using the example files).
(These conda images are always quite big (1.7 GB). Optimised a lot, but don't get it smaller ...)

[bernhard-42]

@bollwyvl Currently the installation works like this (which is already a huge improvement)

conda create -n cq21-jl3 -c conda-forge -c cadquery python=3.8 cadquery
conda activate cq21-jl3
pip install jupyter-cadquery==2.0.0-rc1
jupyter lab

The shortest I could imagine would be (after release of 2.0.0)

conda create -n cq21-jl3 -c conda-forge -c cadquery python=3.8 cadquery jupyter-cadquery=2.0.0
conda activate cq21-jl3
jupyter lab

More importantly. if CadQuery 2.1 is already installed in an environment, then a simple

pip install jupyter-cadquery==2.0.0
jupyter lab

does the trick. Nice!

This JL 3 stuff is so much simpler already. Awesome job!

[bollwyvl]

works is good!

However: re conda-forge: as long as you have a dependency (as you should) on a (bottom-pinned version) of cadquery that doesn't exist on conda-forge, we won't be able to ship your stuff until it's up there. In the end, I have no doubt it will be worth it, as it will be much easier to keep everything in sync... this is not a light build chain...

[bernhard-42]

@bollwyvl You were right, the HTML export is broken. What I saw today, I lost two features by migrating to JL3

• export the pythreejs renderer to HTML via embed_minimal_html
• Jupyter notebook support - which I wanted for using voila (the idea is to have a voila standalone cadquery renderer that visualizes cadquery models sent to the voila app via rest api calls. That would allow to use my Jupyter-cadquery as CAD viewer while developing in any IDE with debugger, ...

Now I need to find ways to get both features back - at least the first feature, since it looks like there is a PR in volia zu a lab app as frontend (not sure it would solve my problem)

Just some feedback on how easy it is to migrate to JL3 ;-)

[tescalada]

I've been working on trying to pull parts of jupyter-cadquery into a flask app so I can publish my parametric models to a webapp. I just figured out why my html has not been rendering. Not sure if its the same issue causing the html renderer to not work here but it might be. I was seeing the message Falling back to https://unpkg.com/ for jupyter_cadquery@2.0.0. Which doesn't exist in unpkg. I don't know why it is trying to pull 2.0.0 but the fix for me was serving 1.0.0 locally so it doesn't try to hit 2.0.0. That made the html rendering work in my webapp.

[bernhard-42]

@tescalada Just release 2.0.0. HTML export now works again. Please look into the example 1-cadquery.ipynb to see how

[bernhard-42]

@bollwyvl Fixed embedding and notebook support and published 2.0.0.
I think the issue came from overwriting the package.json file during the JL2 -> JL3 migration. I lost webpack and other stuff for ipywidgets.

[bollwyvl]

Hm, appears to still be having issues on binder:

the initial issue appears to be jupyter-three.js not being in that folder... is it expected it will marshall that for you?

[bollwyvl]

Binder i used for above:
https://mybinder.org/v2/gh/bernhard-42/jupyter-cadquery/HEAD?urlpath=lab/examples/1-cadquery.ipynb

In looking at the binder environment, I see it installs the rc from pip... perhaps making the binder "hot" (e.g. doing the full build), and still tagging the release, would give better results. Let me know if you're interested, happy to PR!

[bernhard-42]

The threejs stuff is not in the export.html Instead unpkg will be used to download.
So, if you download the export.html from jupyter file system to your laptop and double click it, it works.
The way the embedding works is that if the widgets are missing they are fetched from unpkg.com:

So from the Jupyter UI I think it is a CORS problem to read from unpkg.com

[bernhard-42]

Binder i used for above:
https://mybinder.org/v2/gh/bernhard-42/jupyter-cadquery/HEAD?urlpath=lab/examples/1-cadquery.ipynb

The link I embedded in the Readme.md contains the release version, because I want to control when a new version is used for binder (e.g. I had earlier two binder links, one to the old version and one to a tested dev version).
https://mybinder.org/v2/gh/bernhard-42/jupyter-cadquery/v2.0.0?urlpath=lab&filepath=examples%2Fassemblies%2F1-disk-arm.ipynb

In looking at the binder environment, I see it installs the rc from pip... perhaps making the binder "hot" (e.g. doing the full >build), and still tagging the release, would give better results. Let me know if you're interested, happy to PR!

I am pretty new to binder and the way I did it was the way it worked for me. So happy to learn from you how to improve it.
However, I would prefer to only have tagged versions used for binder, not every HEAD

[bollwyvl]

contains the release version,

Ah, yes, i see that now, must have clicked off a bad link.

prefer to only have tagged versions used for binder

Welp, binder only builds when you ask it to. So keeping the advertised one pointed at a release is great, and should usually stay in cache.

My big argument is usually that having the hot one (but you have to think about building the link a little bit) allows PRs to be evaluated interactively. By having semi-loose pins, e.g. pip >=21, this further allows some ability for the binder to help catch issues during a development cycle.

Another approach to a "release" binder is to have a branch, e.g. binder that pretty much only contains the bare essentials to run, and can be locked down as tight as you want (highly recommend conda-lock): this can be a nice piece of documentation, as well. You can use, e.g. nbgitpuller to pull in the examples, etc. though it can be tricky to build the link up correctly.

[bernhard-42]

@bollwyvl

Happy to help in any way I can!

I know, it wasn't meant like that ... but just wanted to put your attention to jupyter-widgets/pythreejs#359. pythreejs is really awesome, I just feel it could need some more love :-)

[bollwyvl]

Yeah, that's a pretty big ship to turn... I don't know the full impacts of what a large version bump would entail with respect to codegen, etc. but the mouse stuff sounds very compelling. A little slammed right now, trying to get some demos out, but there's always a chance $DAY_JOB might need some new hotness related to geometry that might get me moving in this space. Feel free to try out a PR over there and ping me... Use cases talk, but code walks, and it's much easier to assess big changes in the context of (broken) CI.

[bernhard-42]

Closing for now since Jupyterlab 3 is supported