federated extensions, webpack sharing, deploying/configuring docs #58

bollwyvl · 2021-04-20T22:52:52Z

References

for Support federated extensions #55

Code changes

Adds build from upstreams
better handle setting federated_extensions in the index.html files
try with a simple extension that has schemas, for example: https://github.com/jtpio/jupyterlab-cell-flash
- for now schemas are all bundled into a all.json file on build:
be able to just copy the prebuilt extension assets to lab/extensions
- not have to edit all.json when new federated extensions are installed?
settings service will have to be aware of federated extensions
- dynamically construct the list of all settings in this getAll() method.
a @jupyterlite/top (or site or something) that carries and abstracts some of the boilerplate
- should be able to dedupe the app webpack bundles

User-facing changes

Deployer

Someone who wants to deploy a site would be able to
- at the very leasst on this PR
  - drop some well-formed packages in <app>/extensions (as copied from $share/labextensions)
  - run a .py and/or .js script to update various all.json files

End User

n/a (though load times might be.... different, due to multiple packages)

Backwards-incompatible changes

?

vercel · 2021-04-20T22:52:56Z

This pull request is being automatically deployed with Vercel (learn more).
To see the status of your deployment, click below or on the icon next to each commit.

🔍 Inspect: https://vercel.com/jtpx/jupyterlite/6JKfpTnPYwWaSXeQrpGjdpd8b9Yc
✅ Preview: https://jupyterlite-git-fork-bollwyvl-add-federated-extensions-jtpx.vercel.app

jtpio · 2021-04-22T08:10:46Z

Thanks for getting this started!

jtpio · 2021-04-30T08:39:26Z

Trying this out locally:

with https://github.com/jtpio/jupyterlab-python-file as the test extension
manually copying the prebuilt assets to /lab/extensions

Also changed the structure and boilerplate code so it follows the upstream example a bit more closely: https://github.com/jupyterlab/jupyterlab/blob/master/examples/federated/core_package/webpack.config.js

jtpio · 2021-04-30T08:57:26Z

Next steps:

better handle setting federated_extensions in the index.html files
try with a simple extension that has schemas, for example: https://github.com/jtpio/jupyterlab-cell-flash
for now schemas are all bundled into a all.json file on build:

https://github.com/jtpio/jupyterlite/blob/534111c7c3b9f1dbe04c7b99c9eb7bff187e63a1/app/webpack.config.js#L37-L59

so the lab settings manager can fetch all settings at once:

https://github.com/jtpio/jupyterlite/blob/534111c7c3b9f1dbe04c7b99c9eb7bff187e63a1/packages/settings/src/settings.ts#L37

It would be nice to be able to just copy the prebuilt extension assets to lab/extensions and not have to edit all.json when new federated extensions are installed. Basically a copy should be enough. But then the jupyterlite settings service will have to be aware of federated extensions, and be able to dynamically construct the list of all settings in this getAll() method.

bollwyvl · 2021-04-30T14:48:57Z

Thank you so much for looking into this. I'll check it out soon... the delta from my broken stuff should be enough for me to keep moving forward.

settings service will have to be aware of federated extensions

Yeah, sounds like table stakes for getting this in.

Humans should not have to write anything other than the one JSON file that seeds pageinfo, etc. so this is just another part of that, i guess...

bollwyvl · 2021-04-30T15:11:03Z

Nothing to concretely comment on, but the duplication between classic/lab definitely points to us perhaps wanting a @jupyterlite/top or something that carries and abstracts some of the boilerplate... will have to do some thinking.

jtpio · 2021-04-30T15:18:33Z

but the duplication between classic/lab definitely points to us perhaps wanting a @jupyterlite/top or something that carries and abstracts some of the boilerplate

Yeah that sounds good.

the one JSON file that seeds pageinfo

And for now we could limit the scope to Python driven installs, which sounds the simplest so we could copy straight from $PREFIX/share/jupyter/labextensions to lab/extensions.

bollwyvl · 2021-04-30T15:27:19Z

could limit the scope to Python driven installs

Oh, yes, indeed. Nobody benefits from us building another package format.

So in another nuts-and-bolts, hopefully-testable diagram, I think we want two paths:

Integrate into existing Jupyter-based workflows #41 will be able to fully populate everything, because we can guarantee having a working lab sitting around, but won't need nodejs
for the "plumbing" approach (want to run webpack and stuff), we have a separate directory for py-module-0.0.0.tar.gzs (to cleanly separate from pyolite .whls... also they are half the size), and do the dirty in nodejs to figure out where they are put with data_files, copy them out. It seems preferable this is not done during webpack, though it could start webpack. So basically @jupyterlite/builder.

Both would end up fully realizing in lab/extensions/(<ns?>)/<package>/package.json, and generating whatever metadata slugs we need.

bollwyvl · 2021-04-30T16:28:53Z

Further, I think we have to plan for being able to deploy servlite extensions _as_ federated extensions, and downselect them based on some metadata at setup time... We can't maintain (and make people go use) _another_ build chain, and then figure out where to upload it, etc. It should be possible to work against a local one, however, without packaging, ideally live deployed into the app with "labextension watch"... We can monkey patch is we must, but I think all the switches are available some way or another. However, we don't need to solve that on this pr! Well already be able to do lots of good demos with this, modulo the things we've called out (which I'll hoist to the checklist)

bollwyvl · 2021-05-02T21:41:22Z

jtpio · 2021-05-03T12:13:08Z

Thanks, this is looking great!

However, we don't need to solve that on this pr! Well already be able to do
lots of good demos with this, modulo the things we've called out (which
I'll hoist to the checklist)

Yeah absolutely, need to give some more thoughts to this. Also in the case where third-party packages want to distribute both a serverlite and a "normal jupyterlab extension" together.

So I reckon we add tour and a few other goodies to docs, and see how it goes...

We could put something lightweight into the current docs, however, without checking stuff in... it's really just a question of what are our top-flight selections. Of course, I'm partial to ipydrawio, but it is pretty heavy...

Sounds good 👍 I think we could also have a list of curated extensions that work well, or just some that we like to use.
Generally speaking, it would be nice to always have a functional demo on main that doesn't break too often. So as long as it doesn't disrupt the rest, adding new federated extensions to the default demo website on RTD should be fine.

jtpio · 2021-05-03T12:19:15Z

app/config-utils.js

+/**
+ * TODO: consider better pattern for invocation.
+ */
+await main();


Looks like this might give a top level await error on some browsers.

yeah, i just wanted something that didn't webpack for this stage. as mentioned, we likely want this for "in-app" stuff too, so it probably needs to become a package with an entry point in the top-level webpack.

We also need to do more work on cache-busting, getting that [hash] back into file names, as I've noted on RTD it's pretty easy to get in a bad cache state. there's probably some more stuff we can fiddle with to get more-reproducible chunks.

So top level await seems to be supported by Chromium 90 and Firefox 89.0b7 (developer edition). But not in Firefox 88.

Actually do we need the await here? Or could this be changed to the following (at least for now):

void main();

docs/deploying.md

Co-authored-by: Jeremy Tuloup <jeremy.tuloup@gmail.com>

jtpio · 2021-05-03T14:25:58Z

.eslintrc.js

@@ -45,7 +45,7 @@ module.exports = {
      { avoidEscape: true, allowTemplateLiterals: false }
    ],
    curly: ['error', 'all'],
-    eqeqeq: 'error',
+    eqeqeq: ['error', 'always', { null: 'ignore' }],


Do we still need the null: 'ignore' part? Or was is for testing?

It would be nice to stick to === checks in general, but maybe this was added for a specific reason?

strict equality of null and undefined is not consistent, whereas if you let the browsers Do Their Casting Thing, it comes out correctly... in my experience. Indeed, I prefer the behavior where it complains if you ever try to strict equal null or undefined.

So do you think we should keep that ignore clause, or leave it like it was before?

It might sound like a cosmetic thing mostly, but I was thinking we could try to follow the JupyterLab TypeScript Style Guide to make it easier for folks to switch between code bases and contribute to these projects. It doesn't explicitly say anything about strict equality, but I guess it would try to follow the JupyterLab eslint / prettier config when that makes sense.

jtpio · 2021-05-03T14:35:33Z

packages/settings/src/settings.ts

    name: STORAGE_NAME,
    description: 'Offline Storage for Settings',
    storeName: 'settings',
    version: 1
  });
+
+  export async function getAll(): Promise<{ settings: IPlugin[] }> {


Looks like the settings are not updated on the RTD preview for this PR, for example when switching the theme or moving tabs to the left and right area: https://jupyterlite--58.org.readthedocs.build/en/58/_static/lab/index.html

Although refreshing the page does pick them up. So maybe it's related to the new getAll "merging" logic.

yeah, gross. needs test #6 lol

Yeah absolutely.

Just pushed c0bff5f in the meantime which should fix it.

It adds the network requests back to fetch all.json and the federated extension schemas too. But it would fine to take a look at it again in a separate PR if we want to optimize this.

This should also handle editing settings for a federated extension via the settings editor.

yeah, settings editing works now that the isFederated check is accurate

bollwyvl · 2021-05-03T14:46:08Z

list of curated extensions

That's a whacking good idea. quick sketch:

as we don't yet know what we want it to look like "for real", add a dumb file of:
- map of npm-package-name to
  - full .whl URLs
  - whether we want them in the demo site
  - notes about their deployment
  - maybe a link to any open issues about supporting lite
during the docs build, use that file to
- download and unpack
- write jupyter-lite.json
- build up the docs page
  - read whl metadata to get homepage etc
  - look at the on-disk size to see how much heft we're talking

As for that dumb file:

I don't see this becoming part of the end user tooling, per se
- probably best to stick to stdlib dependencies when possible, which means .json or .ini 😿
but for docs purposes, maybe TOML for pr-ability?
- we already have flit around, which depends on toml
  - even though we won't be able to ship all the packages with it, because data_files 🙀

always have a functional demo on main

No complaints there! Indeed, #6 should be testing that (probably deployed locally with e.g. tornado), not just the empty-and-light-as-possible contents of app which might not be minified, etc.

distribute both a serverlite and a "normal jupyterlab extension" together.

Well... on Jyve, i paid the 🐒 price to make stuff work in both full lab and the offline thing, and it was a huge hassle... but i'd definitely want to be able to use things like jupyterlab/pull-requests with Notebooks for kernelites.

What we can't do is introduce a new package build chain: it's gotta build with labextension build, and end up as another plugin.

I'd see hoisting serverlite-extensions up to jupyter-lite.json and making that something that the tooling can either detect at build time by introspecting a jupyterlite key in package.json, or that a deployer can manually specify.

jtpio · 2021-05-03T15:28:12Z

we already have flit around, which depends on toml

Using flit was quite arbitrary mostly to get the pyolite kernel up and running quickly (and it's simple to use), but we could switch to something else if needed.

What we can't do is introduce a new package build chain: it's gotta build with labextension build, and end up as another plugin.

That would be ideal yes. Just need to check whether we can keep that workflow given that the jupyterlite server app is (for now at least) a separate Lumino application.

bollwyvl · 2021-05-03T15:36:51Z

Using flit was quite arbitrary

I strongly desire being able to ship jupyter tools with flit, if only for the reproducible wheels but we'll be stuck with setup.cfg and setuptools for actually shipping the built applications for the foreseeable... once we're worrying about cache busting and being able to confidently update jupyterlite deploys, this gets pretty clutch...

docs/configuring.md

bollwyvl · 2021-05-03T22:33:00Z

packages/settings/src/settings.ts

+   */
+  export async function getFederated(id: string): Promise<IPlugin | undefined> {
+    const [ext, plugin] = id.split(':');
+    let federated: string[];


oops, i think this comes back as a {name, load, extension, style}... will update...

bollwyvl · 2021-05-03T22:55:22Z

packages/settings/src/settings.ts

  }

  /**
   * Get all the settings
   */
  async getAll(): Promise<{ settings: IPlugin[] }> {
+    return Private.getAll();


so... these will definitely not pick up federated stuff at present...

looked into this a bit... the problem is there is no enumeration of ["@org/pkg:plugin", ...] anywhere in the metadata. When we get to the "for real" automation tools, we may wish to hoist that information as well, as we would have the contents of the canonical schemas directory available, and could make that available to jupyter-config-data.

it could also be that the convention of the directory structure isn't actually enforced... but i think we'll just have to found out once we have the automation to start trying out a bunch of a extensions.

bollwyvl · 2021-05-04T18:58:57Z

Yeah want to circle back on that... Ideally it would be in ts, as part of a single build, with a cache-busting target, as if you get a bad cache of that and things change, it's not going to go well. Then each of the Index.html would also have to point at the cache-busted script, which complicates matters, but it's worth it. Also want to hoist the schema to ts types.

jtpio · 2021-05-04T19:09:50Z

@bollwyvl I just pushed a couple of minor tweaks to:

remove the top level await so it works in the latest stable Firefox. And we can probably add it back later once FF 88 is the new stable
switch back to the previous eslint rule regarding the null checks (just to keep things consistent but could be added later again if needed)

Otherwise I think it looks really good for a first iteration on federated extensions! We can merge it as is and iterate in follow-up PRs and issues.

bollwyvl · 2021-05-04T19:41:54Z

Sure, sounds good if you're feeling it. We should hoist the TBD from this to new issues if not already done... Will look later today

jtpio · 2021-05-04T19:52:31Z

This is really great stuff, thanks again Nick!

vercel bot deployed to Preview April 20, 2021 22:52 View deployment

jtpio mentioned this pull request Apr 29, 2021

Does not work if you start classic first #79

Closed

very broken federated extensions, webpack sharing

07654a4

jtpio force-pushed the add-federated-extensions branch from 84f6972 to 07654a4 Compare April 29, 2021 15:06

jtpio added 2 commits April 30, 2021 10:12

Basic federated extensions in lab

cde1cd6

Load federated extension in classic /tree

a81e167

Update to index.template.js

691c294

jtpio added 2 commits April 30, 2021 11:14

Lint

ef5c70d

Clean up

a49a8c9

bollwyvl added 11 commits May 1, 2021 23:40

add config-utils.js

c2f469b

move config to .jupyter-lite

be5a938

use real json extension

b6d6eda

ensure jsonschema is available

5a2c6c9

remove vanity dot

04ea277

add sphinx-jsonschema

c042047

get schema docs working under test

b8ae1dc

update docs

42ced16

more work on docs

442deb3

add dev command for federated extensions

7405545

federated extension settings

904560a

This was referenced May 3, 2021

Deployment Documentation #67

Closed

Allow overriding plugin settings #81

Closed

bollwyvl marked this pull request as ready for review May 3, 2021 00:21

This was referenced May 3, 2021

API Documentation #66

Closed

Make lab extension work without (full) server extension jupyterlab-contrib/jupyter-videochat#25

Closed

jtpio reviewed May 3, 2021

View reviewed changes

docs/deploying.md Outdated Show resolved Hide resolved

Update docs/deploying.md

565a4ae

Co-authored-by: Jeremy Tuloup <jeremy.tuloup@gmail.com>

jtpio reviewed May 3, 2021

View reviewed changes

docs/configuring.md Show resolved Hide resolved

Handle settings flow

c0bff5f

bollwyvl commented May 3, 2021

View reviewed changes

handle shape of federated_extensions

1bd88e6

bollwyvl commented May 3, 2021

View reviewed changes

Minor tweaks and formatting

f1b08d5

Fix handling of mime extensions in lab

7204d45

jtpio merged commit 373b8aa into jupyterlite:main May 4, 2021

jtpio mentioned this pull request May 4, 2021

Default list of federated extensions on the demo website #83

Closed

jtpio added this to the 0.1.0 milestone May 4, 2021

hbcarlos mentioned this pull request May 5, 2021

Add federated extensions mamba-org/quetz-frontend#67

Merged

5 tasks

jtpio added the enhancement New feature or request label Jun 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

federated extensions, webpack sharing, deploying/configuring docs #58

federated extensions, webpack sharing, deploying/configuring docs #58

bollwyvl commented Apr 20, 2021 •

edited

Loading

vercel bot commented Apr 20, 2021 •

edited

Loading

jtpio commented Apr 22, 2021

jtpio commented Apr 30, 2021

jtpio commented Apr 30, 2021

bollwyvl commented Apr 30, 2021

bollwyvl commented Apr 30, 2021

jtpio commented Apr 30, 2021 •

edited

Loading

bollwyvl commented Apr 30, 2021

bollwyvl commented Apr 30, 2021 via email

bollwyvl commented May 2, 2021

jtpio commented May 3, 2021

jtpio May 3, 2021

bollwyvl May 3, 2021

jtpio May 3, 2021

jtpio May 4, 2021

jtpio May 3, 2021

bollwyvl May 3, 2021 •

edited

Loading

jtpio May 3, 2021

jtpio May 3, 2021

bollwyvl May 3, 2021

jtpio May 3, 2021

jtpio May 3, 2021

bollwyvl May 3, 2021

bollwyvl commented May 3, 2021

jtpio commented May 3, 2021

bollwyvl commented May 3, 2021

bollwyvl May 3, 2021

bollwyvl May 3, 2021

bollwyvl May 4, 2021

bollwyvl commented May 4, 2021 via email

jtpio commented May 4, 2021

bollwyvl commented May 4, 2021 via email

jtpio commented May 4, 2021

federated extensions, webpack sharing, deploying/configuring docs #58

federated extensions, webpack sharing, deploying/configuring docs #58

Conversation

bollwyvl commented Apr 20, 2021 • edited Loading

References

Code changes

User-facing changes

Deployer

End User

Backwards-incompatible changes

vercel bot commented Apr 20, 2021 • edited Loading

jtpio commented Apr 22, 2021

jtpio commented Apr 30, 2021

jtpio commented Apr 30, 2021

bollwyvl commented Apr 30, 2021

bollwyvl commented Apr 30, 2021

jtpio commented Apr 30, 2021 • edited Loading

bollwyvl commented Apr 30, 2021

bollwyvl commented Apr 30, 2021 via email

bollwyvl commented May 2, 2021

jtpio commented May 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bollwyvl May 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bollwyvl commented May 3, 2021

jtpio commented May 3, 2021

bollwyvl commented May 3, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bollwyvl commented May 4, 2021 via email

jtpio commented May 4, 2021

bollwyvl commented May 4, 2021 via email

jtpio commented May 4, 2021

bollwyvl commented Apr 20, 2021 •

edited

Loading

vercel bot commented Apr 20, 2021 •

edited

Loading

jtpio commented Apr 30, 2021 •

edited

Loading

bollwyvl May 3, 2021 •

edited

Loading