5.0 release plan

[krassowski]

This is the current plan for 5.0 release:

• switch to JupyterLab's 4.0 @jupyterlab/lsp package
• turn on more eslint rules, including camelCase CamelCase naming convention? #47

As per feedback, remaining items were moved to 5.1 release plan

[skukhtichev]

Is there a timeline for jupyterlab-lsp 5.0 release?

[krassowski]

There is no timeline yet - it may depend on community contributions.

[LiutongZhou]

Jupyterlab V4.0.2 is released. Looking forward to V4 support.

[sh-shahrokhi]

Could you prioritize " switch to JupyterLab's 4.0 @jupyterlab/lsp package", maybe add other features as time permits?

[krassowski]

I am working on JupyterLab 4.0 support; it requires a substantial rewrite due to a different editor (CodeMirror 6), the @jupyterlab/lsp package which does not know about things, and virtual notebook. I will probably share some version for testing before 5.0 release so that anyone who is impatient can try it out and report bugs, while other things on agenda get addressed (but yes, we can push some things to next release).

[krassowski]

First pre-release supporting JupyterLab 4.0 is now available: v5.0.0a0. There is a number of known issues, including many around completer, which will need fixing upstream and require a new JupyterLab release.

[krassowski]

First beta is out: v5.0.0b0. Please report any bugs you see, this will help accelerate the final release.

[bollwyvl]

Have started conda-forge/jupyter-lsp-feedstock#50, which would be installable from

mamba install -c conda-forge -c conda-forge/label/jupyter_lsp_beta "jupyterlab-lsp==5.0.0b0"

• that new channel will have copies of all of the jupyter*-lsp* packages, can't do anything about it, but isn't going to hurt anything, either

Some packaging thoughts:

• we could look to use some of the approaches from e.g. bqplot to avoid shipping duplicate copies of the labextension static files in the .whl
  • a major release might be an appropriate time to adopt flit or hatch
• we could go further and consider dropping the .js.map files from the .whl
  • have considered some patterns in conda-land to do split packages, e.g. jupyterlab-lsp-with-sourcemaps
    • this could theoretically be done in the .whl level, too, but would be... more complicated

[bollwyvl]

Also, perhaps an npm 5.0.0-beta.0 release would be useful for downstreams to start kicking tires?

[bollwyvl]

Tried kicking all the future tires with this gist on binder.

I had the jupyterlab 4.1.0 alpha in there, but it didn't like it:

Unsatisfied version 4.1.0-alpha.1 from @jupyterlab/application-top of shared singleton module @jupyterlab/lsp (required ^4.0.5)

At least during the pre-release cycle, we should probably allow that case...

[bollwyvl]

Lots of other weirdness underway... maybe we try to carve out some time after the wednesday call to take a look at some things.

[krassowski]

This is because of strictVersion in:

jupyterlab-lsp/packages/jupyterlab-lsp/package.json

Lines 150 to 156 in 98a8794

	"sharedPackages": {
	"@jupyterlab/lsp": {
	"bundled": false,
	"singleton": true,
	"strictVersion": true
	}
	},

I will disable it for 5.0.0b1.

[krassowski]

npm 5.0.0-beta.0 release

this will need bumping version and somewhat stabilising APIs. I am still somewhat tempted to break up features into packages but it is becoming increasingly less likely without resources and with other more pressing deadlines. Anyways, once the completer issues are fixed upstream (and if there are no further issues found in beta1) I think we can publish an RC with npm packages.

[krassowski]

Once #978 is in, the only remaining blocker for RC will be upstream JupyterLab completer work, which had 1 PR merged, 1 in review, and 1 in the works; if we are lucky this could happen in the upcoming week.

This is was the most difficult version transition so far, with many things subtly breaking due to the way the core LSP was merged into JupyterLab. I am glad this is nearly over.

[Pigrenok]

Is it possible to install 5.0.0b1 (or any other pre-release) from repo itself? pip still have only released version. Thanks.

[krassowski]

pip still have only released version.

No, it does not: https://pypi.org/project/jupyterlab-lsp/5.0.0b1/ use -U --pre switches to install latest pre-release. A release candidate is likely to show up by Monday.

[bollwyvl]

Also available on conda-forge:

mamba install -c conda-forge/label/jupyter_lsp_beta -c conda-forge jupyterlab-lsp
# stuff happens ...
# + jupyterlab-lsp                      5.0.0b0  pyhe83ca6f_0        conda-forge/label/jupyter_lsp_beta/noarch      568kB 

[krassowski]

Release candidate is up:

• https://github.com/jupyter-lsp/jupyterlab-lsp/releases/tag/v5.0.0rc0
• https://pypi.org/project/jupyterlab-lsp/5.0.0rc0/
• https://www.npmjs.com/package/@jupyter-lsp/jupyterlab-lsp/v/5.0.0-rc.0

If nothing major comes up I will publish the final release one week from this moment.

[krassowski]

Note: a release blocker has been identified (crash of the browser on editing of large notebooks with magic transculsions, like %%sql due to memory/websocket leak - #988) which is why there was no final release early this week. This is related to the transition of LSP machinery to core JupyterLab chopping off an implicit cache mechanism of webocket connections, see #959. The upstream issues are:

• Virtual document updates may lead to a memory leak jupyterlab/jupyterlab#15144
• Problems extending virtual LSP documents downstream jupyterlab/jupyterlab#15126

There may be a way to workaround it in jupyterlab-lsp by rewriting the update manager, but this defeats the purpose of migrating the code upstream for easier reuse, so ideally the fix would be applied upstream too. The best course of action would probably be developing a fix here and then applying it upstream for 4.1 release as it would likely extend API surface. This might take more time. CC @hbcarlos (for awareness).

Alternatively, we could release 5.0 with transculsions disabled by default. This would lead to worse UX but should avoid the crashes. What do you all think?

[astrojuanlu]

I've been using the rc for some time and works like a charm, maybe disable transculsions (what a word 😄)? But I'm not a user of those, so I don't have the complete picture

[firai]

Since fixing magic transclusions is expected to take some time, I would advocate releasing 5.0 with magic transclusions turned off, and putting it on known regressions and 5.1/5.2 to-do lists. If the jupyterlab-lsp API is expected to be affected by this work (not sure if it will be), maybe put a warning in the docs that this area is WIP (although this may not be critical since the rest of the API docs may not have been updated per #962 anyway).

[SylvainCorlay]

Note: a release blocker has been identified (crash of the browser on editing of large notebooks with magic transculsions, like %%sql due to memory/websocket leak - #988) which is why there was no final release early this week. This is related to the transition of LSP machinery to core JupyterLab chopping off an implicit cache mechanism of webocket connections, see #959. The upstream issues are:

• Virtual document updates may lead to a memory leak jupyterlab/jupyterlab#15144
• Problems extending virtual LSP documents downstream jupyterlab/jupyterlab#15126

There may be a way to workaround it in jupyterlab-lsp by rewriting the update manager, but this defeats the purpose of migrating the code upstream for easier reuse, so ideally the fix would be applied upstream too. The best course of action would probably be developing a fix here and then applying it upstream for 4.1 release as it would likely extend API surface. This might take more time. CC @hbcarlos (for awareness).

Alternatively, we could release 5.0 with transculsions disabled by default. This would lead to worse UX but should avoid the crashes. What do you all think?

We will put resources towards fixing this in the next couple of weeks, hopefully fixing the last outstanding issues.

[krassowski]

Thank you so much @SylvainCorlay! Collaborating with your team on bringing this upstream is amazing and I am grateful for open communication.

If I were to do this again, I would have advocated on moving some features upstream immediately, because the approach that we mutually agreed on - of porting the core machinery to core and only then porting the extension - turned out to be painful because code was not integration tested along the way (but then there was also a moving piece of CodeMirror 6 so I understand that it would be difficult anyways).

I applied a workaround in the extension by force-overwriting a private method from upstream (@jupyterlab/lsp) for now and will release 5.0 with it, but hopefully this will not be needed for the next version :)

[krassowski]

v5.0.0 is now available on npm and PyPI. Thank you all for testing and support!

[bollwyvl]

also from conda-forge.