Support importing pyodide as ESM module

[leopsidom]

🚀 Feature

This is to propose supporting loading pyodide.js as ESM module. Majority of modern browsers support ESM modules via script: can i use module in script. Both chrome and safari support ESM modules in worker script: can
i use module in worker. Firefox is also actively working on supporting ESM modules in web workers: Implement worker modules. They resolved a blocker issue recently: Factor out common ScriptLoadRequest for use in Worker and DOM.

With ESM type module supported, we should be able to import pyodide as follows:

<script type="module">
import { loadPyodide } from "https://cdn.jsdelivr.net/pyodide/v0.19.1/full/pyodide.js"
</script>

And similar syntax in web worker so we can create a module worker as new Worker('webworker.js', { type: 'module' })

Motivation

The main motivation of this feature is so that it can integrate more easily with modern bundling tool like vite.js, which only supports bundling module type web worker. The lack of support on ESM module import means that pyodide.js needs to be manually handled -- manually exposed (if building an npm package from it) and copied when installing the npm package.

Pitch

Basically we would like the following syntax to work, instead of throwing an error:

The requested module 'https://cdn.jsdelivr.net/pyodide/v0.19.1/full/pyodide.js' does not provide an export named 'loadPyodide'

<script type="module">
import { loadPyodide } from "https://cdn.jsdelivr.net/pyodide/v0.19.1/full/pyodide.js"
</script>

And similar syntax in web worker so we can create a module worker as new Worker('webworker.js', { type: 'module' })

One option I think is to expose current pyodide.js as pyodide.umd.js and creates another version as pyodide.es.js that supports ESM module import. And replace importScripts:

pyodide/src/pyodide.js

Line 71 in 2633091

self.importScripts(url);

with esm import syntax. importScripts throws an error when the script is imported as a module type worker.

Alternatives

Use current as it is, and the downside is that we need to manually manage the bundling.

Similar issue: #684

Hope this makes sense :)

[leopsidom]

Oh it seems we already built out esm format here: https://github.com/pyodide/pyodide/blob/main/src/js/rollup.config.js#L40. We just need to fix the issue importing it with ESM module in a web worker here:

pyodide/src/pyodide.js

Line 71 in 2633091

self.importScripts(url);

[leopsidom]

I tried a simple fix in the compat.ts, https://github.com/pyodide/pyodide/blob/main/src/js/compat.ts#L114 with the following:

loadScript = async (url) => {
  try {
    globalThis.importScripts(url);
  } catch (error) {
    return await import(url)
  }
}

It seems to work. Not sure if there's a better way to detect whether a worker is a classic type or module type though.

[hoodmane]

Thanks for investigating @leopsidom. I would be happy to accept a PR that adjusts loadScript in this way. We only use loadScript once (to load pyodide.asm.js) in any case.

[leopsidom]

@hoodmane Cool. I opened a PR here: #2220 with the above fix.

[alexmojaki]

👍 @hangtwenty also ran into this when trying to migrate futurecoder to Vite.

I see #2220 is merged, does that mean this issue is solved?

[hoodmane]

I am not sure. I ran into trouble importing Pyodide from a module webworker the other day, so I am not convinced that it is working. We could leave this open until we get more feedback, or I suppose we could close it and let someone open a new issue if they run into more problems.

[leopsidom]

Currently the ESM target pyodide.mjs is not yet exposed to CDN. We only expose the UMD version pyodide.js. So I think we just need to expose the .mjs version to CDN. At least that's how it looks to me for 0.19.0: https://www.jsdelivr.com/package/npm/pyodide

[hoodmane]

Well the npm has neither of these. That pyodide.js is the file from src/js/pyodide.js not the one you would need to execute. We have the module here: https://pyodide-cdn2.iodide.io/v0.19.1/full/pyodide.mjs but when I tried to import it into a module webworker I got weird errors.

[leopsidom]

I just tried this url. I think the change has not been released yet. I'm getting the same old error:

Module scripts don't support importScripts().

while the error points to the original version of loadScript.

[hoodmane]

Yeah, the change is available in the development version:
https://pyodide-cdn2.iodide.io/dev/full/pyodide.mjs

[leopsidom]

Cool that URL works. I tried to bundle it with vite.js here: : https://stackblitz.com/edit/vitejs-vite-dhqdgo?file=src/main.ts. And it works in ESM format. To see the page in action, you can just run npm run dev in the terminal:

Natively import with new Worker(..., { type: 'module' }) should work as well, which is what vite uses internally. But I think Worker has some contraints on the file mimetype, so you need to serve the worker file with the text/javascript mimetype. I set up a static server with http-server: https://github.com/http-party/http-server given the correct mime type during my testings and it was working.

[alexmojaki]

The changelog for 0.20.0 mentions this issue, is it fully solved now?

[hoodmane]

I think it works with ES6 modules. There are still a lot of problems with webpack, npm, etc, see #2394 #2393, etc. I will optimistically close this, but feel free to reopen or make a new issue if you think there is still a problem.

[pauleveritt]

@leopsidom I'm trying your worker approach in Vite but the build output with fails in the IIFE with unknown variable at the end: (pyodide_mjs)

I'm very interested in the permutations of Pyodide, Vite, Vitest, ESM, module workers. It's a challenge: top-level await, Firefox not doing module workers, etc.

I have a 19-part tutorial series I'm wrapping up. Can I pick your brain on build?

[manzt]

Natively import with new Worker(..., { type: 'module' }) should work as well, which is what vite uses internally. But I think Worker has some contraints on the file mimetype, so you need to serve the worker file with the text/javascript mimetype. I set up a static server with http-server: http-party/http-server given the correct mime type during my testings and it was working.

For development this is fine in Chrome, but Worker type: module isn't wildly supported across browsers. For production, vite bundles the worker code separately.