Fails to load Pyodide from external URL in node

[ryanking13]

🐛 Bug

It fails to load Pyodide when we set indexURL to some external URL in node.

To Reproduce

npm install pyodide@0.22.1

cat > index.mjs << EOF
import { loadPyodide } from "pyodide";

(async () => await loadPyodide({indexURL: "https://cdn.jsdelivr.net/pyodide/v0.22.1/full/"}))();
EOF

node index.mjs

Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/workspace/temp/nodetest/https:/cdn.jsdelivr.net/pyodide/v0.22.1/full/pyodide.asm.js' imported from /workspace/temp/nodetest/node_modules/pyodide/pyodide.mjs

Environment

• Pyodide Version: 0.22.1

Additional context

The problem happens in resolvePath

path.resolve that we use in node.js considers URL as a local filesystem path.

> path = require("path");
> path.resolve(".", "https://example.com")
'/workspace/temp/nodetest/https:/example.com'

[hoodmane]

Well node_resolvePath should return path unchanged if it contains ://. But then there's a problem with __dirname.

[hoodmane]

The problem is here:
https://github.com/emscripten-core/emscripten/blob/main/src/shell.js?plain=1#L223
If I fix that, it successfully fetches pyodide.asm.js but fails to run it with nodeVmMod.runInThisContext with the error:

A dynamic import callback was not specified.

[hoodmane]

Using

nodeVmMod.runInThisContext(result, {async importModuleDynamically(specifier) { return await import(specifier);}})

fixes that error. But now:

Aborted(TypeError: Cannot read properties of undefined (reading 'normalize'))

[hoodmane]

I suppose one option would be to write the url to a file and then import the file, rather than using vm.runInThisContext.

[hoodmane]

We can fix the next error by patching getBinaryPromise to use the ENVIRONMENT_IS_WEB path which fetches the module. But then we get:

Error: ENOENT: no such file or directory, open 'http://0.0.0.0:8003/pyodide.asm.data'
Error: ENOENT: no such file or directory, open 'http://0.0.0.0:8003/repodata.json'

[ryanking13]

Well node_resolvePath should return path unchanged if it contains ://. But then there's a problem with __dirname.

Hmm, I don't get it. node_resolvePath is defined as follows:

nodePath = await import("path");
function node_resolvePath(path: string, base?: string): string {
  return nodePath.resolve(base || ".", path);
}

and it doesn't seems to return path unchanged.

[hoodmane]

Yeah that code should be changed to the following:

function node_resolvePath(path: string, base?: string): string {
  if(path.includes("://")) {
    return path;
  }
  return nodePath.resolve(base || ".", path);
}

But there are lots of problems. I think to fix this we most likely need to patch Emscripten in a couple of places.

[ryanking13]

Oh, thanks. I misunderstood what you said. I naively thought that fixing node_resolvePath would work, but it seems like the problem is more complicated than that.

[hoodmane]

Yeah there just seem to be a lot of places where code expects a URL if in browser and a path if in node. I think we could upstream fixes where needed though.

[savakarrohan]

I am still facing this issue, any updates for it v0.25.0