Federated bundle (for JLab>=3) and source bundle (for JLlab<3) in 1 pypi package

[maartenbreddels]

For many python widget libraries, we put the js source tarball in $prefix/share/jupyter/lab/extensions. JLab 1 and 2 pick this up, and rebuild. For JLab 3, we want to put the Federated bundles into $prefix/share/jupyter/labextensions, e.g.:

• Use Jupyterlab3/Federated Extension System glue-viz/bqplot-image-gl#35
• Add support for Jupyterlab 3 bqplot/bqplot#1222

However, if we want our new distributed python package to work for both JLab versions, not requiring a rebuild on 3, asking for a rebuild on 1 and 2. Is there a way to do this?

[jasongrout]

Can you check in the latest jlab 3 prerelease? Install jlab 3.0rc6 and one of your widgets packages that has the jlab 2 bundle. Does jlab 3 correctly recognize it can't use the bundle, and ignore it?

[jasongrout]

The other way should work fine, since jlab 1 and 2 don't know anything about the new location we put federated extensions, so they should just ignore it. Doesn't hurt to test, of course.

[maartenbreddels]

This was on jlab 3.0rc6 (as reported by @mariobuikhuizen).

Does jlab 3 correctly recognize it can't use the bundle, and ignore it?

How should it recognize that the bundle should be ignored? Because there is no dependency on jupyterlab in any of our projects, and they should be compatible with 1, 2 and 3.

[jasongrout]

How should it recognize that the bundle should be ignored? Because there is no dependency on jupyterlab in any of our projects, and they should be compatible with 1, 2 and 3.

Good point.

This is tricky - .tar.gz extensions are incorporated at jlab build time (federated extensions are ignored then), but federated extensions are incoporated at runtime (extensions requiring a rebuild are ignored then, unless they are already built in, in which case they are already in the browser).

Another tricky thing here - in doing jupyter labextension list, federated extensions take precedence over built extensions, I believe. We should make sure that reflects what actually happens if the same extension is both built in and federated.

Perhaps one thing we can do here is to check, before building an extension into jlab, if the same federated extension is available (i.e., the same npm package name). If it is, do not build it in, preferring the federated extension.

[jasongrout]

Perhaps one thing we can do here is to check, before building an extension into jlab, if the same federated extension is available (i.e., the same npm package name). If it is, do not build it in, preferring the federated extension.

This doesn't solve the problem of building it in at some point, then later installing the federated version.

[maartenbreddels]

Indeed, could there be a race condition, that even during a pip install, the .tar.gz gets seen, but not yet the federated bundle, and the user gets asked to rebuild?

I also wonder if this enabled multiple versions of the same widget library. If so, we should probably exclude building when both the package name and the version matches (exactly or semver?).

[jasongrout]

From the dev meeting:

1. We want to support both federated and raw extensions
2. Federated extensions take precedence
3. Make it very clear to users in labextension install, build, and the build check that it is ignoring the raw extension in favor of the federated extension
4. Runtime warning if a federated extension overshadows a raw extension
5. If we have both a federated and raw extension that are the same version (or if federated is same or newer?), silently pick the federated extension over the raw extension and ignore the raw extension.

[jtpio]

Documenting the process of adding the federated bundle to an existing extension here for reference. So we have more details with a concrete example.

• Using JupyterLab 3.0.0rc10
• The JupyterLab extension is: https://github.com/mamba-org/gator (also known as jupyter_conda)
• It bundles both the tgz and the federated bundle for lab 3. After a pip install .:

$ jupyter labextension list
JupyterLab v3.0.0rc10
/home/jtp/miniforge3/envs/gator/share/jupyter/labextensions
        @mamba-org/gator-lab v2.0.0-rc.0 enabled OK (python, mamba_gator)

Other labextensions (built into JupyterLab)
   app dir: /home/jtp/miniforge3/envs/gator/share/jupyter/lab


Build recommended, please run `jupyter lab build`:
    @mamba-org/gator-lab needs to be included in build

• Opening JupyterLab triggers the popup:

• The federated extension is functional ✔️
• The extension manager lists both copies:

---

It should actually be possible to recreate a smaller example by starting from the 3.0 branch of the cookiecutter: https://github.com/jupyterlab/extension-cookiecutter-ts/tree/3.0

And adding the tgz to the data files like here: https://github.com/jupyterlab/extension-cookiecutter-ts/blob/fb827d3f438f62a1dc7566800f6b37d0d2a1fe8b/%7B%7Bcookiecutter.extension_name%7D%7D/setup.py#L34

[maartenbreddels]

Hi all,

jupyterlab 3 is getting amazing :), but i'm afraid this is not fixed in v3.0.0.rc13.

To reproduce (a bit elaborate because of an install issue with that branch):

$ mamba create -y -n jlab3mixed nodejs jupyterlab=3 ipywidgets jupyterlab_widgets -c conda-forge/label/jupyterlab_rc -c conda-forge/label/jupyterlab_server_rc -c conda-forge/label/jupyterlab_widgets_rc -c conda-forge
$ conda activate jlab3mixed 
$ pip install jupyterlab==3.0.0rc13
$ cd tmp
$ git clone https://github.com/maartenbreddels/ipydatawidgets
$ cd ipydatawidgets
$ git checkout jlab_fed_bundle
$ python setup.py bdist_wheel
$ pip install dist/ipydatawidgets-4.1.1.dev0-py2.py3-none-any.whl
$ jupyter lab 

This gives:

[C 2020-12-08 17:21:05.790 ServerApp] 
    
    To access the server, open this file in a browser:
        file:///Users/maartenbreddels/Library/Jupyter/runtime/jpserver-26766-open.html
    Or copy and paste one of these URLs:
        http://localhost:8889/lab?token=x
     or http://127.0.0.1:8889/lab?token=x
[W 2020-12-08 17:21:10.762 LabApp] Build recommended
[W 2020-12-08 17:21:10.762 LabApp] jupyterlab-datawidgets needs to be included in build

[vidartf]

To add some more details, I'm now testing the updated widget package prior to a pre-release (vidartf/ipydatawidgets#43), in various conditions:

Clean lab3 env

If I install it into a clean lab 3 env:

pip install --pre jupyterlab jupyterlab-widgets
pip install <path to a built dist>

then I get greeted with a "jupyter-datawidgets needs to be included in the build" dialog at startup (since it installs both a source and pre-built extension). Everything works well other than that, since the extension is running off the prebuilt version.

Running jupyter labextension list gives me:

> jupyter labextension list
JupyterLab v3.0.0rc13
<env dir>\share\jupyter\labextensions
        jupyterlab-datawidgets v6.3.0 enabled ok
        @jupyter-widgets/jupyterlab-manager v3.0.0-alpha.2 enabled ok

Other labextensions (built into JupyterLab)
   app dir: <env dir>\share\jupyter\lab

And if I run jupyter lab build, as prompted the widgets then stop to work:

Plugin 'jupyter.extensions.datawidgets' failed to activate.
Error: No provider for: jupyter.extensions.jupyterWidgetRegistry

Updating from a lab 2 env:

If I again start with a clean env, then install lab2 and some extensions, and the update lab:

pip install jupyterlab
pip install <path to build dist>
jupyter labextension install @jupyter-widgets/jupyterlab-manager   # also builds lab
# Things work in this state with lab 2

pip install --upgrade --pre jupyterlab jupyterlab-widgets

then I get greeted with a "jupyter-datawidgets needs to be included in the build" dialog at startup, but things work. And if I rebuild and reload, things again stop to work (but with a different error compared to the other case):

Could not instantiate widget
Error: Could not create a model.
Exception opening new comm
Uncaught (in promise) Error: Module jupyter-datawidgets, semver range ^5.1.0 is not registered as a widget module

So in conclusion, distributing both a pre-built and source extension at the same time seems messy with the current state of things.

[vidartf]

PS: The upgrade path might get fixed by jupyter labextension update --all once the final release is out, as the updater doesn't really take pre-releases into account (I thought that it should, but it seems that it does not).

[vidartf]

So, while #9489 resolves the problem with the build check recommending a build (which is good 👍 ), there is still an issue:

• Create a clean env
• pip install --pre jupyterlab jupyterlab-widgets
• pip install <path to datawidgets master>
• Now the build check will no longer recommend that you need to build, but if you do (either because you think it will be better, or because you install another source extension), you will still fail like before:

  Plugin 'jupyter.extensions.datawidgets' failed to activate.
  Error: No provider for: jupyter.extensions.jupyterWidgetRegistry
  

[vidartf]

To clarify, the situation that you have is that you have:

• an extension that is both available as a pre-built and source extension,jupyterlab-datawidgets
• that depends on an extension that is present as a pre-built extension only, @jupyter-widgets/jupyterlab-manager

and the lab extension system is not able to fulfill the dependency, even though both extensions are loaded.

[jasongrout]

Those two scenarios should work since rc11 or so (with the right configuration for sharedPackages...). I'll check with your packages too to see if we messed something up.

[jasongrout]

First, confirming that things work without rebuilding, I set up a clean environment with nodejs and python and pip<20.3 installed from conda-forge, then do:

• pip install --pre jupyterlab jupyterlab_widgets
• pip install ipywidgets
• pip install <path to https://github.com/vidartf/ipydatawidgets clone, master branch>

I open an example notebook and everything seems fine:

and labextension list seems fine as well (no rebuild suggested):

% jupyter labextension list                                                                                       master
JupyterLab v3.0.0rc14
/var/folders/ln/5623mk393v77n5l3rk2x88h00000gp/T/tmpenv.iqBVgLHB/env/share/jupyter/labextensions
        jupyterlab-datawidgets v6.3.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.0-alpha.2 enabled OK

Other labextensions (built into JupyterLab)
   app dir: /var/folders/ln/5623mk393v77n5l3rk2x88h00000gp/T/tmpenv.iqBVgLHB/env/share/jupyter/lab

Then I run jupyter lab build, and indeed I see the js errors: Plugin 'jupyter.extensions.datawidgets' failed to activate. and Error: No provider for: jupyter.extensions.jupyterWidgetRegistry.

[jasongrout]

Continuing debugging in the above scenario...

It turns out that @jupyter-widgets/base is built into jlab now, but arguably should not be (or at least if it is, it should be a shared module, so it deduplicates with the prebuilt jupyterlab_widgets extension).

It also looks like the ipydatawidgets extension is built into JupyterLab and is the actual extension run (i.e., it is not disabled, like it should be since it also exists as a prebuilt extension), which begs the question:

A. If a prebuilt and source extension bundle exist, should the source extension bundle be built into jupyterlab even though we know it is shadowed by the prebuilt extension and should be disabled/ignored, or should it be ignored in the build and never built in?

[vidartf]

I think we should try to handle the situation where a package is both included in a build from a source extension, and also present as a prebuilt extension. Otherwise, we would also need to recommend builds to remove it if it is found from both sources. Doable, but it sounds painful.

[jasongrout]

After a lot of craziness, it turns out that my yarn had cached an old version of ipydatawidgets with the same version number which was messing everything up. Doing a yarn cache clean, then jupyter lab clean and jupyter lab build makes ipydatawidgets work fine.

[jasongrout]

(Moving ipydatawidgets to a dev version number after the last release would have likely fixed the issue I saw)

[jasongrout]

I think we should try to handle the situation where a package is both included in a build from a source extension, and also present as a prebuilt extension. Otherwise, we would also need to recommend builds to remove it if it is found from both sources. Doable, but it sounds painful.

What we should make sure, though, is that one of these is disabled, even if it is present in the js in the system.

[vidartf]

To expand on what Jason has already has commented, I think I have seen a similar problem on 2.x sometimes: If you are developing an extension, and install it, and then make some further changes and install it again, it can sometimes use the first version you installed. Cache cleans have seemed unreliable in these cases, but that might be system dependent. I've often ended up writing a little script that bumps the version before installing, just to ensure I don't end up with a cache issue. So a developer quality of life issue, not a release issue.

[jasongrout]

Let's open a new issue if you want to investigate the dev issue more. Thanks!