Update `jupyterlite` dependencies and docs #1001

jtpio · 2023-03-10T13:07:47Z

References

Update the dependencies for the jupyterlite metapackage.

Follow-up to #994.

Part of #993

This will be a follow-up to jupyterlite/pyodide-kernel#23.

Code changes

Update jupyterlite-core dependency
Depend on jupyterlite-pyodide-kernel
Streamline more CI with jupyterlab/maintainer-tools/.github/actions/base-setup@v1 (remove some logic already handled by the action)
Test on Python 3.8
Update docs to mention jupyterlite-core and how to install the Pyodide kernel

User-facing changes

This will ensure end users depending on jupyterlite will still get the Pyodide kernel by default for now.

Backwards-incompatible changes

None

github-actions · 2023-03-10T13:07:59Z

👈 Try it on ReadTheDocs

jtpio · 2023-03-10T13:08:12Z

Pending jupyterlite/pyodide-kernel#23 and a release of jupyterlite-pyodide-kernel.

jtpio · 2023-03-10T14:24:01Z

Since we're requiring Python 3.8+ in jupyterlite-pyodide-kernel (https://github.com/jupyterlite/pyodide-kernel/blob/c3f1d74a48510f4fa39f52b6c04312e946090be7/pyproject.toml#L25), I would say we should just to the same here as well.

Or just test on 3.8 instead of 3.7.

jtpio · 2023-03-13T18:01:33Z

OK I think this should be good to go now.

Planning to make a new 0.1.0b19 release of both jupyterlite-core and jupyterlite from the repo after the merge.

Some follow-ups items for later:

another pass on the docs
remove the 0.1.0-beta temporary tag and revert Point docs to 0.1.0-beta in the README for now #997
finish updating the downstreams to depend on jupyterlite-core
Simplify the dev setup #684

bollwyvl · 2023-03-13T20:18:46Z

dodo.py

-            actions=[U.do(*args, cwd=cwd)],
-            file_dep=file_dep,
+            name="py:jupyterlite",
+            actions=[U.do(*metapackage_args, cwd=P.ROOT)],


removing any existing file_dep is a recipe for having to re-run stuff over and over again with no reason, and probalby out of order. ideally, we'd be adding more accurate file_dep and targets, when at all possible.

yeah probably we can add it back. This is likely a leftover of fighting with the doit tasks

Actually I don't know, this task is just here so we can simply install the packages in editable mode...

Right: but it would at least depend on:

pyproject.toml (changing entry_points, data_files, and a raft of other things invalidates an editable install)

anything indirectly consumed by pyproject.toml (e.g. _version.py or wherever the source of truth is these days)

I'm planning to have another look at this as part of #684

docs/howto/extensions/frontend.md

docs/howto/configure/simple_extensions.md

bollwyvl · 2023-03-13T20:23:40Z

.github/actions/build-dist/action.yml

@@ -20,7 +20,6 @@ runs:

    - name: Install
      shell: bash
-      if: steps.cache-node-modules.outputs.cache-hit != 'true'


😿 caching node_modules (provided .eslint and .prettier don't muck about in there, as they do by default for some reason) has been really stable, and saves an appreciable real amount of time... meanwhile, without a way to clean out the yarn cache, it just gets bigger, and misses a lot.

it does opens us up for e.g. the svgo issues, which seem to happen anyway when the runners change their apt deps, i guess.

I don't think this one was used anyway, the jupyterlab/maintainer-tools/.github/actions/base-setup action above should already handle caching.

For svgo maybe we should remove it from the CI / dodo tasks. I don't know if we need to optimize the images all the time, this could be done occasionally when needed. And also setting up a fresh new env almost always leaves the git repo in a dirty state with some icon files being modified.

should already handle caching.

... of the ever-expanding, never-pruned ~/.cache/yarn, which then still requires compiling binaries from source, sometimes, and invoking the solver.

for svgo

My point was "any crazy node_modules," but svgo in specific could be replaced with the stable, but slightly less efficient pure-python scour. But maybe more closed-form, and would cause less thrash.

Anything that "only gets run sometimes,".... doesn't.

Anyway that's off topic, I opened #1004 to discuss this more.

bollwyvl · 2023-03-13T20:25:56Z

docs/howto/configure/advanced/extensions.md

@@ -82,7 +79,7 @@ If detected, [`libarchive-c`](https://pypi.org/project/libarchive-c) will be use
 better performance, especially when working with archives with many/large assets,
 espcially [pyodide](../../python/pyodide.md).

-If not `libarchive-c` is not detected, python's built-in `zipfile` and `tarfile` modules
+If `libarchive-c` is not detected, Python's built-in `zipfile` and `tarfile` modules


we should move away from libarchive-c: outside of conda/mamba, it is a pain to install... i don't know what to replace it with, though... the whole idea is to be able to use binary tools for vetted performance and security, but every one of them carries complexity risks.

Let's track that in an issue?

bollwyvl · 2023-03-13T20:26:40Z

py/jupyterlite/pyproject.toml

-    # TODO
-    # "jupyterlite-core[lab]",
-    # "jupyterlite-pyodide-kernel",
+    "jupyterlite-pyodide-kernel >=0.0.4",


why not also the compatible lab?

you mean as a regular plain dependency?

Not sure we need to add it for now since it was not the case previously with the jupyterlite package? Also there is already a [lab] extra.

I guess we could indeed include more dependencies as mentioned in #993 (like ipywidgets). Maybe we can first see how it goes and add them later.

My thinking here: the "happy path" of one of the most complex dependency issues, relying on the users' package manager of choice to make sys.prefix whole, will start getting less happy as extensions don't install/solve correctly because of jupyterlab 4... and [extras] are basically broken.

A future jupyterlab-core can then depend on jupyterlab >=4,<4.1, even when we maintain the 3.5 compatible one.

I like widgets for this same reason: we can support ipywidgets >=7.6,<8, but there's no guarantee all future versions will have compatible jupyterlab_widgets releases, and getting kernels up to supporting widgets represents a huge amount of effort (right next to arbitrary python packages, and iteratively reducing friction there).

Let's track that in a separate issue about which package to include in jupyterlite by default.

Opened #1003

This reverts commit 5003ba9.

This reverts commit ce86104.

This reverts commit 1e9c3ae.

jtpio · 2023-03-15T07:28:47Z

Getting this one in to unblock other work and we can address follow-ups in other PRs.

jtpio added the maintenance label Mar 10, 2023

jtpio added this to the 0.1.0 milestone Mar 10, 2023

jtpio added 12 commits March 10, 2023 14:53

Update jupyterlite dependencies

0827c87

Add lower bound for jupyterlite-pyodide-kernel

658564d

Update to 0.0.4

c98e980

Remove --no-index for now

6ccf10b

Also in ui tests

f792168

Follow same approach as other deps

776a138

try upgrade

81d65f9

Test on 3.8 for now

53cba20

fix syntax

e2cd372

drop flag

fbaf42b

no deps

6288dc4

drop lab extra for now

e5d4beb

jtpio force-pushed the update-packages branch from 7250ec3 to e5d4beb Compare March 10, 2023 14:53

jtpio added 12 commits March 10, 2023 14:59

add pkginfo for now

afba301

add core

795e999

try without jupyterlite for now

e00c7d1

fix step

03aedd4

uncomment in dodo

46c07ca

skip jupyterlite for now

4a6597c

(tmp) comment -m invocation for now

0dfebae

Update LITE_INVOCATIONS

4101cf5

update pip cmd

fd958cf

Streamline base-setup

f3b23cc

try more cleanups

9ad52f5

reenable jupyterlite on docs

1727566

jtpio added 7 commits March 13, 2023 15:43

cleanup

caafe17

dodo

6420fdc

...

28d9897

more dodo voodoo

d43822a

.

477ec38

more docs update

ba8df8e

cleanup

a4d6b5c

jtpio marked this pull request as ready for review March 13, 2023 18:01

jtpio requested a review from bollwyvl March 13, 2023 18:01

jtpio added 2 commits March 13, 2023 18:24

try reenable tests in docs

60c4915

clean leftovers

1b0701a

bollwyvl reviewed Mar 13, 2023

View reviewed changes

docs/howto/extensions/frontend.md Outdated Show resolved Hide resolved

bollwyvl reviewed Mar 13, 2023

View reviewed changes

docs/howto/configure/simple_extensions.md Outdated Show resolved Hide resolved

bollwyvl reviewed Mar 13, 2023

View reviewed changes

jtpio added 5 commits March 13, 2023 20:36

docs edits

5003ba9

try leaving the dev task as it was before

1e9c3ae

Revert "docs edits"

ce86104

This reverts commit 5003ba9.

Revert "Revert "docs edits""

3658873

This reverts commit ce86104.

Revert "try leaving the dev task as it was before"

8d8a0e9

This reverts commit 1e9c3ae.

This was referenced Mar 14, 2023

List of packages to include with the jupyterlite metapackage #1003

Open

Depend on jupyterlite-core jupyterlite/xeus-python-demo#12

Merged

jtpio merged commit ff49d65 into jupyterlite:main Mar 15, 2023
17 checks passed

jtpio deleted the update-packages branch March 15, 2023 07:29

This was referenced Mar 15, 2023

Default docs back to latest #1009

Merged

Require Python 3.8, test on Python 3.11 #1016

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update `jupyterlite` dependencies and docs #1001

Update `jupyterlite` dependencies and docs #1001

jtpio commented Mar 10, 2023 •

edited

github-actions bot commented Mar 10, 2023

jtpio commented Mar 10, 2023

jtpio commented Mar 10, 2023 •

edited

jtpio commented Mar 13, 2023 •

edited

bollwyvl Mar 13, 2023 •

edited

jtpio Mar 13, 2023

jtpio Mar 13, 2023

bollwyvl Mar 13, 2023

jtpio Mar 15, 2023

bollwyvl Mar 13, 2023

jtpio Mar 13, 2023

jtpio Mar 13, 2023

bollwyvl Mar 13, 2023

jtpio Mar 14, 2023

bollwyvl Mar 13, 2023

jtpio Mar 14, 2023

bollwyvl Mar 13, 2023

jtpio Mar 13, 2023

jtpio Mar 13, 2023

bollwyvl Mar 13, 2023

jtpio Mar 14, 2023

jtpio Mar 14, 2023

jtpio commented Mar 15, 2023 •

edited

Update jupyterlite dependencies and docs #1001

Update jupyterlite dependencies and docs #1001

Conversation

jtpio commented Mar 10, 2023 • edited

References

Code changes

User-facing changes

Backwards-incompatible changes

github-actions bot commented Mar 10, 2023

jtpio commented Mar 10, 2023

jtpio commented Mar 10, 2023 • edited

jtpio commented Mar 13, 2023 • edited

bollwyvl Mar 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jtpio commented Mar 15, 2023 • edited

Update `jupyterlite` dependencies and docs #1001

Update `jupyterlite` dependencies and docs #1001

jtpio commented Mar 10, 2023 •

edited

jtpio commented Mar 10, 2023 •

edited

jtpio commented Mar 13, 2023 •

edited

bollwyvl Mar 13, 2023 •

edited

jtpio commented Mar 15, 2023 •

edited