Update external libraries and document their versions

[dclause]

I would like to introduce a PR to conditionnally load the external libraries in dev mode because of:
"Vue.js is detected on this page. Devtools inspection is not available because it's in production mode or explicitly disabled by the author."

To do so, I have a question on external library version: which versions are they ?
I can see no mention on external library version in the README.md (which would be a nice addition to read the appropriate documentation). Using that, I would be able to provide the vue.global.prod.js equivalent dev file. I guess one version of this:
https://unpkg.com/browse/vue@3.2.47/dist/vue.global.js

---

Task list:

• (1) First we should update all libraries as promised in vue/quasar/tailwind update process #318.
• (2) For transparency we should use a naming scheme including the version number. --> No, we keep the names without version number.
• (3) We should improve our routing paths to include the version numbers. This way we avoid issues with browsers having outdated cached versions.
• (4) And we should mention the packaged versions in the README.md and/or on nicegui.io.
• (5) Update Three.js addons manually.
• (6) Fix joystick example (broken after nipple.js upgrade)
• (7) How to resolve layout inconstistency? --> Fix website.

Maybe later:

• (8) Use ESM dependencies like Mermaid 10.0 and Three.js?

[falkoschindler]

That's a good question. A few months ago, when getting NiceGUI 1.0 ready, I might just have copied the currently stable versions without keeping track of their version numbers. But the upcoming 1.2 release is a good opportunity to get this straight.

[falkoschindler]

@rodja How do we get the current NiceGUI version at runtime?

[rodja]

@falkoschindler Maybe with pkg_resources: python-poetry/poetry#273 (comment)?

[rodja]

... or __version__ = importlib.metadata.version(__package__ or __name__) as described in python-poetry/poetry#273 (comment)

[dclause]

Hi @falkoschindler,

Could you clarify the difference between poin t2 and 3 ?
If the externallibrary is named library.js: on point 2, you suggest renaming it library-3.2.1.js or equivalent I guess. That make point 3 obsolete I suppose. Point 3 is usually done with cache key as a parameter : library.js?v=3.2.1

Point 3 is a preferred solution I would advocate here because that is a scheme adopted in the web world and CDNs, tools, etc.. are used to it. It would also be easier to automate the dependencies update process if one day that becomes scripted.

Also, how about adding a dev version unminified ?
That would let you use vue devtools for instance https://chrome.google.com/webstore/detail/vuejs-devtools/nhdogjmejiglipccpnnnanhbledajbpd or debug your quasar component on front.

That would require an extra key in ui.run and ui.run_wtih may dev: True and False by default to bind the proper version I suppose.

[falkoschindler]

@dclause Yes, with point 2 I primarily thought about the local file system, while point 3 deals with the request API.
Sure, a pattern like library.js?v=3.2.1 would be possible. But do we store the JS file as library.js? How do we know its version? That's why I thought storing it as library-3.2.1.js would nicely encode its version - independent of point 3.

Actually we thought about changing the current API from:

• https://nicegui.io/_nicegui/static/socket.io.min.js
• https://nicegui.io/_nicegui/dependencies/0/ag-grid-community.min.js
• https://nicegui.io/_nicegui/components/aggrid

to

• https://nicegui.io/_nicegui/1.2.0/static/socket.io.min.js
• https://nicegui.io/_nicegui/1.2.0/dependencies/0/ag-grid-community.min.js
• https://nicegui.io/_nicegui/1.2.0/components/aggrid

Maybe we also need additional library version numbers. But the point is, we need the NiceGUI version to ensure everyone uses the correct version of custom components like aggrid in this example.

[rodja]

Instead of https://nicegui.io/_nicegui/1.2.0/components/aggrid I also like @dclause suggestion of using the schema https://nicegui.io/_nicegui/components/aggrid?v=1.2.0. As @falkoschindler noted it may be simpler to use the NiceGUI version instead of the library version...

[falkoschindler]

Getting the NiceGUI version

Using pkg_resources seems to be discouraged.
Therefore I'll try using importlib.metadata.version, although Python 3.7 seems to require importlib_metadata.version.

Cache-proof API (point 3)

Especially for 3rd-party dependencies like socket.io.min.js it might be confusing to append the NiceGUI version https://nicegui.io/_nicegui/static/socket.io.min.js?v=1.2.0. Instead we prefer inserting it like https://nicegui.io/_nicegui/1.2.0/static/socket.io.min.js.

Using the NiceGUI version instead of the library version has the advantage of less manual work when updating the libraries. The NiceGUI version is incremented automatically and the locally stored libraries are upgraded using a script I'm writing right now.

Local storage (point 2)

When we don't include the library version in the API (point 3), it is easier to store the files without it as well. Then we can mount the static directory without the need of adding or removing version numbers when looking for files.

[dclause]

I did not see your last point: indeed having a script to update the libraries would be nice, and rationalize their inclusion and storage.

In the meantime, I have opened #542 with stuffs from this issue. It is far from anything ready but I wanted to showcase my idea with the production / development mode files.

[falkoschindler]

I started to automatically update all dependencies on the branch dependencies. Besides being pretty cumbersome to find all relevant CDNs and the current versions of each package, there are two mayor issues remaining:

• Mermaid is ESM only since version 10.0. We are currently on 9.x.
• Three.js will be ESM only after version 0.160.0. We are currently on 0.150.1.
• Three.js's addons CSS2DRenderer, CSS3DRenderer, OrbitControls, and STLLoader are already ESM only. For NiceGUI 1.0 I manually modified the modules to be compatible with CJS, but this is impractical for every upgrade.

These are no showstoppers. But if we promise to upgrade all libraries with every minor version change, we should try to do it thoroughly. The next opportunity might be months away. So it would be very interesting to find out how to include these ESM packages.

[falkoschindler]

It gets worse: After updating all packages the layout is pretty off (most prominently the colored pseudo-windows buttons). I'm starting to wonder how we can update packages at all if such problems occur.

[dclause]

I may be wrong but only some minor difference are visible right ? Namely only the padding of elements in the 'windows style' taskbar right ?

That is probably some small adjustements to do in the CSS nicegui.css file. Maybe a stronger CSS selector or explicit padding / margin properties or something that kind. It should be minor though as long as the updates are for minor version bumping.
Updating dependencies always are risky on some edge cases but as to be done.

Speaking of that, do you plan also updating the python dependencies ? I have for instance a compatibility issue with nicegui 1.1.11 and fastAPI 0.94.1 on my project.

As of ESM modules, I am not sure to understand the issue. ES6 module is supported by most modern browsers https://caniuse.com/es6-module
Are you planning a compatibility with browsers so old it does not have it ?
Then you can polyfill it with something like https://www.npmjs.com/package/es-module-shims
Am I missing something ?

Lastly: will the update script be included in the project ?

[dclause]

Just check: vue3 does only support modern browsers anyway, so that is the whole premise of nicegui fundamental I guess here: https://vuejs.org/about/faq.html#what-browsers-does-vue-support

[falkoschindler]

Well, I thought if I notice any visual difference, then there could be larger ones depending on the user code. nicegui.css didn't change, so the cause is probably in Quasar or Tailwind. Of course, fixing the website is not a problem. But we don't want every NiceGUI user to fix his layout after upgrading NiceGUI.

I'm not sure about Python dependencies. Maybe we should discuss it on a separate thread.

My concern about ES6 modules is primarily based on my lack of JS experience: How do we need to modify the import mechanism (mainly in index.html) to load ES6 modules like https://unpkg.com/browse/three@0.150.1/build/three.module.js? And how to handle libraries like https://cdn.jsdelivr.net/npm/mermaid@10.0.2/dist/ which are fragmented into a thousand files? Again: That's probably solvable, but maybe not on a Friday night. 😉

Yes, the update script lives here (at the moment): https://github.com/zauberzeug/nicegui/blob/dependencies/fetch_dependencies.py

[falkoschindler]

I discovered a pretty fundamental difference between both Tailwind versions:

ui.label('Hello World').classes('text-red-400 text-green-500')

On main branch this label is green as it should be, since the green color class overwrites the red one.

But after the update the label is red. The new Tailwind seems to sort classes alphabetically. It also causes the colored window buttons to have a default gap-4 instead of gap-1. In my opinion this is a hugely breaking change.

I'll probably should do some research to find out more about this behavior. Maybe we can configure it.

[falkoschindler]

Ok, the answer to this confusion is given here: tailwindlabs/tailwindcss#10603
Basically the effective class order was already nondeterministic before the update and using contradicting classes was always dangerous and discouraged. Now the order is deterministic such that the result does not change when, e.g., moving the element around.

As an example, the following labels are both green before the update, and both red afterwards:

ui.label('Hello').classes('text-red-400 text-green-500')
ui.label('World').classes('text-green-500 text-red-400')

So I guess the new behavior is better. We should live with it, fix the website, and clearly indicate this change in the release note.

[falkoschindler]

I wonder if we could use npm to automatically fetch all dependencies. 🤔
But I still can't see how to bring a package from the node_modules folder into NiceGUI.

[falkoschindler]

I think in f571ff1 I found a generic solution for the new Tailwind behavior:

We used to set classes like column items-start gap-4 to UI elements like ui.column. When the user additionally sets .classes('gap-1') it's hard to tell which class will win, since all Tailwind classes are rearranged based on a complicated algorithm.

My new approach introduces custom NiceGUI classes like nicegui-column. In nicegui.css we define the default style for such a class, e.g. display: flex; flex-direction: column; align-items: flex-start; gap: 1rem. Importantly, this class does not compete with the later applied Tailwind class gap-1. So Tailwind can simply translate gap-1 and apply it on top of the nicegui-column class.

As far as I can tell, our website looks good again. We still need to inform the users about the Tailwind upgrade. But the impact should be much less dramatic.

Last thought: Browsers need to reload the nicegui.css file to get the new class definitions. But we already planned to introduce the NiceGUI version number in the request path.

[dclause]

Hi !
I have been following this thread and your comment since friday but could not find the time to answser before (also following my own experiments which took some time).

I don't know how to adress and answer all the subjects in this topic in a coherent way, so I will adress them in the order of my experimentations. Let me order them by numbers:

1. How to manage dependencies ?
Really this is an interesting and also philosophical question here: is NiceGUI a python project that happen to have some JS, a JS project that happen to have some python, or an hybrid project ?

My own opinion, NiceGUI is a Python project, aiming at Python developpers. The reasons are:
-> it does not require a JS stack to run (node, npm, etc...)
-> the APIs simplifies everything regarding web or js related stuffs in the project. It is made to be as python-only as possible

That means I would not recommand to add NPM and Node as a requirement for the project. Yet it does not mean we can't use NPM for dependencies :)

First, let me highlight a few stuffs:
-> python dependencies are not included in the project, but are handled via pip install. Having such a command for external (js / css) dependencies would be ok too for me
-> we can install dependencies from NPM but using an external script of our own
-> we can either include the JS dependencies in the repo or not (that means release package process can add the dependencies so they are here when you run pip install nicegui or download a packaged release)
-> we can provide a script to let people install the dependencies. So the contributing process would add after pip install to python install.py for instance.
-> dependencies are mixed on the element/lib folder . Instead, it could be arranged on logicial folders.

At the moment, I will provide a npm.py script I have been working on that is intended to update the dependencies version using NPM as a single source of install. It them moves the dependencies to an appropriate sub-folder of their install path (element/lib or static folder).

2. How to handle ESM libraries ?
On both this and this comments you expressed your concern on how to include ESM libraies. NiceGUI already treats every dependencies as EcmaScript modules: in index.html the import is done in a a <script type="module"> and uses the import syntax.
BUT it is true that ESM modules usually requires an import from syntax. That means:
-> to be able to differentiate between ESM / non ESM module for the proper syntax
-> how can we exactly know the proper syntax for that import ? The name of the variable to destructure is dependency dependent. We can't choose a generic one. It has to be:
import mermaid from ...
import { CSS2DRenderer } from ...
etc...

I have no real answser here, but considering point 3, I do feel like anyway we could make distinction between component js files (currently using register_component, for instance the aggrid.js file) and the external library dependency (for instance ag-grid-community.min.js) that is depends on. I forcing an explicit "import" statement", for instance in the associated js component file can solve both this and the point 4.

3. Loading dependencies:
Many dependencies, especially as they move to ESM, breaks themselves into many many different files (compare mermaid 9 and 10 ^^). Serving those files from NiceGUI server has different issues:
-> we need a mechanism to allow browser to cache the files, hence we need to serve a cache header with an infinite expirity date
-> we need a mechanism to change the URLs based on the version to allow the cache to rebuild. I guess serving the dependencies suffixed with v=1.2.0 (the version of nicegui not the one from the dependency) would solve the deal, because dependencies version can only change when nicegui version changes

Also (I don't know really what others are doing with niceGUI): it seems that you (at Zauberzeug) use niceGUI in robots context (like with rosys). I happen to do it too. NiceGUI is the UI of the python code running my robot. In my case, my code runs on a raspberryPI within the robot, and, by nature of my robot, it is connected to the web. Therefore, serving dependencies from the niceGUI server is actually a bottleneck compared to serving from external CDNs. My client browser's network would be in charge of downloading the assets, rather then my robot server to serve it. Plus, the browser could parallelize those calls much more because they come from different domains. Of course that can't always be the case: your robots may be offline. Hence, it could make sense to provide a CDN option for external dependencies.

4. Optimize the dependencies served
Currently, all components are registered in the ui.py file therefore are 'read by python' (not sure how to word that!). And the register_component is in the component file, outside the class. Therefore also always called. That makes all dependencies of all components always discovered and included.
Take the 'menu and tabs' example at python examples/menu_and_tabs/main.py and run it: this is a very small and empty example. Yet is loads 1.5Mo of compressed JS (5+Mo uncompressed) which is useless. The whole highchart and the whole three.js libraries for instance, mermaid and tween, everything.
I know there is the "exclude" key in ui.run() to handle that, but that is not a proper option:
-> I have to change that everytime I use a new component in my project
-> I have to add new components excluded when they are added to new nicegui versions
-> If I use the "scene" component on one of my hundred page app, I will have to remove it from the exclusion and load the threeJS library in all pages just because on of them needs.
All this makes the exclude key not viable to me.

Here too, I guess I could POC some solution.
-> Maybe if the import statement to external libraries is part of the component (either py or js) itself (as suggested for point 2).
-> Maybe if in the renderRecursively method we keep track of what is rendered
Then we could include only the libraries for what is needed.

5. Allow external library debug and development
I have already discussed this in previous comments: that is the ability to use either the dev or prod version of a library. Not sure if worth for all libraries or just the core onces. Vue.JS for sure would be the must have.

[falkoschindler]

Very, very interesting thoughts, @dclause!

But this issue blew up quite a bit. To be honest: It was originally about serving dev-dependencies and we hijacked it for the whole 1.2.0-dependency upgrade process (because answering your question about the current dependency versions required to get this straight first).

Let me try to address the points you mentioned and find a new home for them:

1. Managing dependencies

   To be clear: I also would use npm only for us NiceGUI developers when updating the dependencies in the static and lib folders. Requiring the users to use npm would not be nice at all.

   I think we should open a new discussion for this topic since it is not crucial and somewhat independent of the current 1.2.0 update.

2. Handle ESM libraries

   I'd love to find a clean solution for it. It would help with point 1 (managing dependencies) and allow using more modern libraries like Mermaid 10.0.

   I think this deserves a separate discussion as well.
   Or should we combine it with point 1?

3. Loading dependencies

   Caching should be solved with task (3) from the task list above.

   Requesting dependencies from the NiceGUI app (Python, e.g. on the robot) or from the internet (some common CDN) is a whole new topic. Currently we try to serve everything from Python to allow working offline and to avoid GDPR issues (General Data Protection Regulation). But allowing to load dependencies from an online CDN would probably be an interesting opt-in feature. JustPy has a NO_INTERNET flag. We could introduce a ui.run(internet=True) parameter. But this is also something for another discussion.

4. Optimize dependencies served

   That's a good point! We already started to think about this here:
   Generalize on-demand dependency loading from chart.js #241
   I'd suggest to continue the discussion there.

5. Dev/prod mode

   Because this issue is already so big, I'd suggest to discuss and develop the dev/prod mode over here:
   Add external libraries development mode. #542

@dclause Would you aggree to open new discussions for points 1-3 and continue discussing point 4 and 5 on the mentioned tickets? I would then concentrate on completing the task list at the top to finalize milestone 1.2.0 for the next release.

[dclause]

@falkoschindler I guess this ticket originally was me trying to understand dependency management because I had those points in mind. But it goes beyond 1.2.0 for sure.
Point 1 is already done on my side. I will provide a PR but outside 1.2.0 milestone.
Point 2 would be needed, unless you agree to descope major version update of mermaid (and others dependencies) and stick with minor update for now until say 1.3.0 ? Point 2 does not find a solution for me that does not solves point 4 at the same time as I have a common idea for both. I will work on a proof of concept.
All other points are clearly out of 1.2.0 milestone scope indeed.

Sorry to bother you with so much considerations and extra work !

[falkoschindler]

@dclause No worries! Your considerations are always welcome. Its extra work worth spending. 😉

Regarding Memaid: I restricted fetch_dependencies.py to version Mermaid 9.x to avoid the trouble with handling the 10.x ESM package. That should be good enough for 1.2.0. All other packages are up to date.

Ok, so you will cover point 1 in a PR, points 2 and 4 in a combined POC, and point 5 in the already proposed PR #524.
Then I will open a new discussion for point 3 (online/offline mode): #564.

[falkoschindler]

The dependencies for NiceGUI 1.2.0 are:

Quasar 2.11.8
Vue 3.2.47
Socket.io 4.6.1
Tailwind CSS 3.2.6
Tween.js 18.6.4
Plotly.js 2.18.2
AG Grid 29.1.0
NippleJS 0.10.1
Mermaid 9.4.3
Highcharts 10.3.3
Three.js 0.150.1

[dclause]

Quick question : any reason plotly was not updated to 2.20.0 ? I created PR #580 to fix.

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'series') error is sent for the chart.py component. I created PR #581 to fix.

[falkoschindler]

Let's discuss these issues over there. See #580 (comment) and #581 (comment).