Run in node.js

[mdboom]

I don't really understand the implications of this or whether it's possible, but it might be handy for testing.

[gabrielfreire]

Hi @mdboom , i made an attempt to load pyodide.asm.wasm on nodejs but no success so far, i installed a nice npm package for the boilerplate npm install webassembly

the code looks like this

const path = require("path");
const wa = require("webassembly");
const wasmPath = path.join(__dirname, '/wasm/pyodide.asm.wasm');
const memory = new WebAssembly.Memory({ initial: 8192, maximum: 8192 });
const table = new WebAssembly.Table({ initial: 16384, element: "anyfunc" });
const importObj = { memory: memory, table: table };
wa.load(wasmPath, { imports: importObj }).then((module) => {
    console.log(module.exports);
}).catch((err) => {
    console.log(`some error: ${err}`);
});

i'm getting this error

LinkError: Import #4 module="env" function="DYNAMICTOP_PTR" error: global import must be a number

can't find a solution for this, not sure what's causing it, maybe you have some idea?

[mdboom]

I wouldn't expect this one to be easy, but it would be really cool if we can find a solution ;)

There are a bunch of things beyond just the Memory and Table that need to be set up to import the WebAssembly. In general, I wouldn't expect to be able to run emscripten-generated code directly like this. These variables are set up in pyodide.asm.js, and get passed to us in the Module.instantiateWasm callback in pyodide.js.

In general, I would look at the code in pyodide.js to see how all this gets bootstrapped and try to emulate that in the node environment.

Even then, I think there are probably some browser-specific things that emscripten is generating here that may break things in node. You may have to recompile pyodide with different settings. I would look at the emscripten docs for info on porting wasm from browser to node in general.

[gabrielfreire]

Would you have the pyodide.asm.js prettified version
i started adding stuff manually but i think it's too much stuff

const importObj = {
    global: {
        NaN: NaN,
        Infinity: Infinity
    },
    env: {
        memory: memory, 
        table: table, 
        tableBase: 0, 
        memoryBase: 0,
        DYNAMICTOP_PTR: 2042576,
        tempDoublePtr: 2041040,
        STACKTOP: 2042592,
        STACK_MAX: 7285472,
        gb: 1024,
        fb: 0,
        abort: function(cb) {},
        enlargeMemory: function(){},
        getTotalMemory: function(){}
    }
};

i could just copy from a pretty pyodide.asm.js

Edit: don't mind, i managed to prettify here

[gabrielfreire]

did you write pyodide.asm.js or it was generated?

[mdboom]

pyodide.asm.js is generated by emscripten. It contains a lot of the emscripten standard library and pyodide.asm.wasm couldn't work without it.

[gabrielfreire]

hmm ok

[gabrielfreire]

i'm almost there
My code

const path = require('path');
const fs = require('fs');
const wa = require('webassembly');
const pyodidejs = require('./src/wasm/pyodide.asm.js');
const wasmPath = path.join(__dirname, '/src/wasm/pyodide.asm.wasm');
const baseURL = path.join(__dirname, '/src/wasm/');
let pyodide;
function loadLanguage() {
    return new Promise((resolve, reject) => {
        let Module = {};
        fetch_node(wasmPath).then((buffer) => buffer.arrayBuffer()).then((arrayBuffer) => {
            let wasm_promise = WebAssembly.compile(arrayBuffer);
            Module.instantiateWasm = (info, receiveInstance) => {
                wasm_promise
                .then(module => WebAssembly.instantiate(module, info))
                .then(instance => receiveInstance(instance));
                return {};
            };
            Module.filePackagePrefixURL = baseURL;
            Module.postRun = () => {
                Module = null;
                console.log('will resolve'); // never gets here
                resolve();
            };
            pyodide = pyodidejs(Module);
        })
    });
}

async function init() {
    await loadLanguage();
    console.log(pyodide.runPython);
    console.log('Python loaded');
}

function fetch_node(file) {
    return new Promise((resolve, reject) => 
    fs.readFile(file, (err, data) => err ? reject(err) : resolve({ arrayBuffer: () => data })));
}
init();

got this error

Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000000 (most recent call first):
exception thrown: abort(). Build with -s ASSERTIONS=1 for more info.

let me know if you have any ideas

[mdboom]

Great progress!

That looks like it's not finding the Python standard library (all the .py files that make the Python interpreter work). That gets shipped to the browser (or node) in the pyodide.asm.data file, with the metadata (what files go where) in pyodide.asm.data.js. If I recall correctly, you have to load pyodide.asm.data.js before pyodide.asm.js or bad things happen.

[gabrielfreire]

Hey @mdboom , there must be an option to generate pyodide.asm.js/pyodide.asm.data.js to target node.js those files are loaded with methods that depends on the browser.
i went to the source to remove some error throw '''s when there is no browser and override a lot of file system methods to be able to download the python standard library i.e FS.createPath() FS.createReadFile() etc...
I managed to download the whole library but there is an internal method called addOnPreRun(cb) that gets called in the end and loads the library files dynamically, this method receives a callback but this callback is not exposed in Modules so i couldn't override it, the callback is using the browser fetch method too and its not possible to override.

it's been a pretty long journey today but i believe i've overridden the maximum things i could to make some stuff work in NodeJS. there must be an option some where to generate asm.js and asm.data.js that can work there, i don't know.

[gabrielfreire]

This is how the code looks like so far

https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/test.js ...
to be able to download the python standard library files you need to uncomment
line 98 // Module.calledRun = true; otherwise it will throw an error before getting to Module.postRun()

pyodide.asm.data.js is a little bit different as well, just added support to NodeJS
https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/pyodide.asm.data.js

[gabrielfreire]

Wooooooooooooooohhh i did it... i'll push to github and send you the link for the final code

[mdboom]

That's great news! It's really nice to have those details worked out.

Are you interested in working that into a pull request against Pyodide? We would need to find a way to work these changes into the build system -- either by applying the right compiler flags to emscripten to generate the right things, or possibly contributing changes to emscripten itself to emit cross-browser-and-node-compatible code.

[gabrielfreire]

Yes, i still need to figure out how to load packages, but i can write python in nodejs already, which is a BIG win.
Unfortunately i had to manually add nodejs support inside pyodide.asm.data.js, same thing for numpy.js (the .js files for the packages have the same structure as pyodide.asm.data.js, they're just there to load the dependencies).

Maybe there is a flag on emscripten to support nodejs for asm.data.js files.

pyodide.asm.js is ok though, it supports multiple environments.

when i have everything tidy up and loadPackage working i can put together a PR

by the way, have you seen my PR on iodide-example, i wrote mic recording support on the notebook

[mdboom]

Awesome!

[gabrielfreire]

It's working, i managed to load python and numpy in NodeJS.

The main class: https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/PyodideNode/PyodideNode.js

i test on: https://github.com/gabrielfreire/neuralnet.js/blob/wasm-nodejs/test.js

That's awesome. Minor support for NodeJS was required on pyodide.asm.data.js and numpy.js files, the optimal scenario would be to generate those files with this support out of the box, but in any case, it's not hard to do that by hand, just a few lines of code on the loadPackage function and some other few lines in other places.

I thought we would need to download the whole python library but there is no need for that, i think it gets loaded in the webassembly memory, i'm still reading the docs to understand webassembly better, i'm very new to this technology but i think it's very impressive.

[rth]

Very nice work on this!

Maybe there is a flag on emscripten to support nodejs for asm.data.js files.

According to the emscripten docs, at least emcc should generate code that also runs on node by default, cf for instance, https://github.com/kripken/emscripten/blob/6cc4c664e4db2d0e4172c298d4ec88519bce1997/src/settings.js#L321-L328

As to pyodide.asm.data.js, packaged/generated by file_packager.py, I couldn't see anything in the code or its docs that says that it's not suitable for nodejs.

override a lot of file system methods to be able to download the python standard library i.e FS.createPath() FS.createReadFile() etc...

If I understand the docs correctly, in browser the MEMFS system would be used while in node it would be NODEFS. In the above PyodideNode.js, there are things like,

var fs = require('fs');

which talks to the node FS directly, while IMO to be environment independent it should be done via emscripten. Looking at pyodide.asm.js, there are things like,

      if (ENVIRONMENT_IS_NODE) {
        scriptDirectory = __dirname + "/";
        var nodeFS;
        var nodePath;
        [...]

so some node specific code seems to be included.

You mentioned it before, but could you give an example of code that should be adapted for node? I couldn't find anything that looks like FS.createPath() or FS.createReadFile() mentioned above in the generated folder except for Module["FS_createPath"](...). Though I may be missing something and I know very little about node :)

[gabrielfreire]

Hi @rth , thank you. pyodide.asm.data.js comes only with support for browsers to download the library, the same goes for numpy.js or any other package, on my repository if you take a look at pyodide.asm.data.js i had to include Module inside the node process process['Module'] = Module and pass this Module to the var Module in the 1st line.

var Module = typeof Module !== "undefined" ? Module : typeof process.Module !== 'undefined' ? process.Module : {};

there is no problem with pyodide.asm.js, it comes with Nodejs support out of the box.

this is an example of code modification included in pyodide.asm.data.js and numpy.js

var fetch = require('isomorphic-fetch'); // <-- for external resources
var loadPackage = function(metadata) {
    var PACKAGE_PATH;
    var PACKAGE_NAME = "build/pyodide.asm.data";
    var REMOTE_PACKAGE_BASE = "pyodide.asm.data";
    if (typeof Module["locateFilePackage"] === "function" && !Module["locateFile"]) {
      Module["locateFile"] = Module["locateFilePackage"];
      err("warning: you defined Module.locateFilePackage, that has been renamed to Module.locateFile (using your locateFilePackage for now)")
    }
    var REMOTE_PACKAGE_NAME = Module["locateFile"] ? Module["locateFile"](REMOTE_PACKAGE_BASE, "") : REMOTE_PACKAGE_BASE;
    var REMOTE_PACKAGE_SIZE = metadata.remote_package_size;
    var PACKAGE_UUID = metadata.package_uuid;
    console.log(REMOTE_PACKAGE_NAME)
    function fetchRemotePackage(packageName, packageSize, callback, errback) {
      function fetch_node(file) { // <-- for local resources
        var fs = require('fs');
        return new Promise((resolve, reject) => 
        fs.readFile(file, (err, data) => err ? reject(err) : resolve({ arrayBuffer: () => data })));
      }
      fetch(packageName).then((buffer) => buffer.buffer()).then((packageData) => {
          if (!Module.dataFileDownloads) Module.dataFileDownloads = {};
          Module.dataFileDownloads[packageName] = {
              loaded: packageSize,
              total: packageSize
          }
          var total = 0;
          var loaded = 0;
          var num = 0;
          for (var download in Module.dataFileDownloads) {
              var data = Module.dataFileDownloads[download];
              total += data.total;
              loaded += data.loaded;
              num++
          }
          total = Math.ceil(total * Module.expectedDataFileDownloads / num);
          console.log(`Downloaded ${packageName} data... (${total}/${total})`);
          callback(packageData);
          if (Module["setStatus"]) Module["setStatus"]("Downloading data... (" + total + "/" + total + ")");
      }).catch((err) => {
          console.error(`Something wrong happened ${err}`);
          throw new Error(`Something wrong happened ${err}`);
      });
    }
.
.
.
.

i had to replace some arrayBuffer checking because the one that existed before was just not working!

function processPackageData(arrayBuffer) {
        if(!arrayBuffer) throw "No input to processPackageData";
        Module.finishedDataFileDownloads++;
        if(!arrayBuffer instanceof ArrayBuffer) arrayBuffer = arrayBuffer.buffer ? arrayBuffer.buffer : null; // <- this is better
        if(!arrayBuffer) throw "bad input to processPackageData";

i believe this is all i had to by hand to make pyodide work on Node.js, of course it looks easy now but i spent hours and hours debugging those modules to figure out what had to be done hehe, i almost gave up =D

Conclusion

i believe that var loadPackage should have nodejs support out of the box, today it throws a big red error if you're not in a browser, i believe this should be enought

[rth]

Thanks for your response @gabrielfreire !

The question is how we can integrate your valuable contributions into pyodide :)

As far as I understand the code in numpy.js etc is generated by emscripten's file_packager.py: essentially it does some string templating (cf. e.g. here). So to do the changes you propose automatically we would need to patch that file. To avoid rebuilding emsdk for each modification, a first solution could be to just copy file_packager.py from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py,

Makefile:FILEPACKAGER=$(PYODIDE_ROOT)/emsdk/emsdk/emscripten/tag-1.38.10/tools/file_packager.py
tools/buildpkg.py:        Path(os.environ['EMSCRIPTEN']) / 'tools' / 'file_packager.py',

Once we have this working, we can propose these modifications upstream in emscripten (or possibly it might be useful to open an issue in emscripten to start with). There is a number of open issues about node js in emscripten github repo (cf e.g. emscripten-core/emscripten#5237).

I think a first step might be to just use load packages from some remote locations same as it happens in the browser.

I was also wondering how to test this. One possibility could be to create another Selenium instance alongside Chrome/Firefox here, that would interact with Node JS via the Selenium Webdriver API. This would allow to run all tests also in NodeJS without much effort. However, I'm not sure this can work in practice: one possibility I saw for Node+Webdriver API was webdriver.io but I'm not sure how to call it from the Selenium-Python package (cf its documentation). Unless testing in that way would defeat the purpose of running in Node.js?

[rth]

a first solution could be to copy file_packager.py from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py,

@gabrielfreire Would you be interested in opening a PR for it? (If you think this is the right approach to apply your changes automatically).

[gabrielfreire]

a first solution could be to just copy file_packager.py from emscripten version 1.38.10 to somewhere in pyodide, add the the desired changes there, then update the path in following files to use this new file_packager.py,

I really like this solution, i believe it is better to have another file_packager.py for pyodide's needs since maybe supporting Nodejs might not be on emsdk's Road map, plus they are very packed with issues and PRs.

I think a first step might be to just use load packages from some remote locations same as it happens in the browser.

Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js browser code for node.js code, which is basically what i did.

This would allow to run all tests also in NodeJS without much effort

i wasn't familiar on how you guys wrote the tests, i took a quick look and it's all python using selenium, if you want to re-use those tests for node.js i'm not sure how you would call it from python, the first thing that comes to my mind is ZeroMQ or RabbitMQ but i believe it should be pretty easy to re-write in pure node.js using karma, mocha or something like that.

Would you be interested in opening a PR for it?

Well, i was working with files generated by @mdboom so i don't have 'pyodide' or any emsdk pre-requisite setup, i think i'll try to setup everything in a Docker machine, for this project, should be easy. Let me know if you already have a Docker configuration for this.

Once i have everything setup i'll apply the changes and generate something to see if it works well, then i open a PR.

[gabrielfreire]

Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js browser code for node.js code, which is basically what i did.

i think you mean, AFTER the new file_packager.py, right ? hehe... yes i agree

[rth]

Well, this could be a good idea, but for any new package we would have to manually replace a package.data.js browser code for node.js code, which is basically what i did.

I guess I was hoping we could arrive at a data.js that might work in both cases (possibly by detecting some environment variables specific to Node JS and adding some code specific to that case). Also I wonder if somewhere in emscripten code base there isn't already some abstraction layer to do part of that (e.g. it does to it for file system).

Let me know if you already have a Docker configuration for this

Not as far as I'm aware, but you could adapt the .circleci/config.yml for that, which lists necessary instructions to setup the environement with the circleci/python:3.7.0-stretch-browsers image.

[gabrielfreire]

Not as far as I'm aware, but you could adapt the .circleci/config.yml for that, which lists necessary instructions to setup the enthronement with the circleci/python:3.7.0-stretch-browsers image.

Great!

Also I wonder if somewhere in emscripten code base there isn't already some abstraction layer to do part of that (e.g. it does to it for file system).

It is strange indeed that it does some environment check for the file system but not for the data.js files.

these lines make sure you will have an error thrown if you're not in a browser

    var PACKAGE_PATH;
    if (typeof window === 'object') {
      PACKAGE_PATH = window['encodeURIComponent'](window.location.pathname.toString().substring(0, window.location.pathname.toString().lastIndexOf('/')) + '/');
    } else if (typeof location !== 'undefined') {
      // worker
      PACKAGE_PATH = encodeURIComponent(location.pathname.toString().substring(0, location.pathname.toString().lastIndexOf('/')) + '/');
    } else {
      throw 'using preloaded data can only be done on a web page, web worker or in nodejs';
    }

A possible change would be something like this

    var PACKAGE_PATH;
    if (typeof window === 'object') {
      PACKAGE_PATH = window['encodeURIComponent'](window.location.pathname.toString().substring(0, window.location.pathname.toString().lastIndexOf('/')) + '/');
      console.warn('Browser environment'); // BROWSER
    } else if (typeof location !== 'undefined') {
      // worker
      PACKAGE_PATH = encodeURIComponent(location.pathname.toString().substring(0, location.pathname.toString().lastIndexOf('/')) + '/');
      console.warn('Web worker environment'); // WEB WORKER
    } else if (typeof process === "object" && typeof require === "function") {
      console.warn('Node environment'); // NODE
    } else {
      throw 'using preloaded data can only be done on a web page, web worker or in nodejs';
    }

the above code is me already working on some changes for file_packager.py.

[gabrielfreire]

@rth , i'm getting a strange error with the new pyodide.asm.js, the one i have locally works well

exception thrown: ReferenceError: window is not defined,ReferenceError: window is not defined
    at _hiwire_get_global (C:\Users\gabriel.freire\Documents\workspace\neuralnet.js\PyodideNode\pyodide.asm.js:1552:36)
    at wasm-function[312]:290
    at wasm-function[10266]:10
    at wasm-function[488]:348
    at wasm-function[2631]:32923
    at wasm-function[14870]:7
    at wasm-function[2626]:3888
    at wasm-function[2625]:31
    at wasm-function[2587]:676
    at wasm-function[11339]:7

It gets fixed when i change this function

 function _hiwire_get_global(idname) {
    var jsname = UTF8ToString(idname);
    return Module.hiwire_new_value(window[jsname])
  }

to

 function _hiwire_get_global(idname) {
    var jsname = UTF8ToString(idname);
    return Module.hiwire_new_value(jsname)
  }

but this change could brake something else so i don't think this is the problem because the old pyodide.asm.js has the same function

[mdboom]

This is great work!

I think it makes sense to have a local copy of file_packager.py for now, and try to update it so the generated code works in both the browser and node. I think we definitely want to have "universal" builds that would work in all contexts.

I can help get that merged back upstream into emscripten if there are any delays.

As for the references to window: I think we'll just need an abstraction that uses window in the browser and global in Node. That should keep things explicit and hopefully not break anything.

As for testing, ideally we'd find a way to use the same tests in all contexts. We could try writing some tests in a Node-based test harness (rather than py.test) so we could run code in-process in Node itself, and then use webdriver.io to run the same code in the browsers. If that works, we would then have to port all of the tests to the new style. An alternative is to stick with what we have, but call out to Node -- you shouldn't have to use anything fancy like zeromq -- node can be communicated with through Unix pipes.

[nomeata]

What happened to this exciting development? Is running pyodide under nodejs possible/supported these days?

[rth]

See the latest status in #183 (comment) which is still up to date.

[lhl2617]

#183 is now closed

Nodejs support would be a huge plus for many electron/node apps depending on Python applications--examples include VSCode extensions that no longer need to bundle a whole Python installation and what not to enable features.

Is there any more information about this? Glad to jump in to help if there is.

[hoodmane]

The most recent discussion about related stuff was in #1477. The strict mode issues in #1477 have been dealt with and I think we exclusively use globalThis to access globals, so some of the basic issues have been addressed. We also have been working toward having less code with better code quality. (I think all of our Javascript code runs in strict mode right now, including all the emscripten glue code.)

That said, I don't think any of the current contributors has much knowledge of or experience with any of these platforms (node, deno, electron), and there is a limited amount of developer energy. So basically we've been resolving issues that are known blockers for node integration, but we haven't been directly trying to get it working ourselves.

An important step would be just to try to run Pyodide on your favored platform and open issues when you run into trouble. We are very open to including changes to increase the range of platforms supported. Once we have basic support, if we added at least some basic node tests to the CI it would upgrade node support from an afterthought to more of a first-class feature.

Is there any more information about this? Glad to jump in to help if there is.

I can answer specific questions about how the Pyodide runtime works. I'm not sure what sort of information you need. Help is very welcome.

[rth]

With the last few changes in #1691 using Pyodide in Node.js with local files should work.

I'm looking for input on what you would like to see in the Pyodide npm package (related discussion in #328 and #1403), particularly when running Pyodide in Node.js.

1. We could install something like node-fetch, to add the missing fetch API in Node and share mostly the same code base for loading Pyodide and PyPi/micropip packages. The issue is unlike in the browser those downloads will then not be cached locally which is not good.

2. For instance in https://www.npmjs.com/package/@pyodide/pyodide all build artifacts are included in the node package. This has the advantage of not having to deal with version compatibility between packages and the npm library, however it results in quite large downloads. Which would in addition, I imagine, not be very useful for people integrating this npm package into a frontend app that would still use the CDN for packages. Also this does not address the question of caching PyPi packages in Node.

   • 2b. A variation of the above is to upload each Pyodide package individually and let npm do the dependency resolution. It's something that we could explore in the future, but for now I don't think it would solve any of the issues we currently have.

3. The npm package contains just the JS+TypeScript code. When used in Node, downloads of Pyodide and PyPi packages are cached locally with npm.im/make-fetch-happen (or similar) and it's also possible to load those packages from the local file system. This would require dealing with version compatibility between Pyodide packages and the npm package though.

Feedback welcome. For now I'm considering going with option 3, unless there are objections.

The other thing that's not clear to me, is how to combine in the same NPM package the use case a) of frontend app development and producing some .js files with rollup b) running Pyodide in Node. Say we need some arbitrary package for running in Node, and add it to the dependencies. Is this potentially an issue for use case a) or it wouldn't matter?

cc @hoodmane @dmondev @teonbrooks @bollwyvl @jtpio @rebornix

[hoodmane]

how to combine in the same NPM package the use case a) of frontend app development and producing some .js files with rollup b) running Pyodide in Node. Say we need some arbitrary package for running in Node, and add it to the dependencies

Yeah I would be interested to know the answer to this, my impression was that this is inconvenient but I have only ever used node for front end stuff.

Here is one possible solution:
https://stackoverflow.com/a/61283419/1942152

[bollwyvl]

For my highly-biased purposes (entirely informed by jupyterlite), I could care less whether pyodide works with nodejs... aside from:

• available via normal means e.g. npmjs.com and assorted mirrors
• reproducibly built, to the extent possible (hard with custom toolchain)
• easily manageable with yarn <2
• imports well into typescript >=4.2
• works painlessly, is tree-shakeable with webpack 5

I am, however, interested in being able to run these in wasmtime and wasmer backends, but haven't even begun to look into it.

[teonbrooks]

we've actually been working on migrating our Python requirement in our Electron app from native Python to Pyodide (makebrainwaves/BrainWaves#157). We've currently done this by downloading the entire versioned zip, and serve the files from a utils folder with the app (added to our webpack config).

our biggest requirements have been that:

• it works in a webworker
• integrate with our build process (based on the electron-react-boilerplate)
• uses yarn
• supports ts

for our package dependencies, we don't need all the supported packages but we haven't had a need to prune it.
related question: is there a way to serve wheels locally to micropip? I could imagine a scenario where i could require additional python packages for our app but wouldn't necessarily want to go through the build process to make its .js/.data version for wasm.

cc: @jdpigeon can you think of anything else we would need from a node-compatible pyodide?

[rth]

Thanks for the feedback @bollwyvl @teonbrooks! It's helpful to have your input on these different use cases. We likely make an alpha release for the npm package in the near future, and it will be more clear then if there are any issues for your use cases.

[eleanorjboyd]

Hi, we've been investigating how to make Pyodide and VS Code work, and allowing Pyodide to run in Node.js would be very helpful since VS Code is based on Node. It would be great to have the the ability to use Pyodide with VS Code through Node support, not only for us but also for our users. Thanks!

[lhl2617]

@eleanorjboyd just a heads up that VSCode is Electron (which bundles Node); this (I assume) should not cause any issues except if the final Pyodide build for NodeJS uses native NodeJS modules (which should not be the case, otherwise Pyodide shouldn't run on Web at all!)

(VSCode-relevant discussion below)
I however wonder if there is separate discussion for bundling Pyodide in VSCode and exposing it in some sort of API to extension devs (since Pyodide is large). I currently maintain a VSCode extension that depends on Python as well, so if the Pyodide instance in VSCode can be shared via an API, it would be ideal. Another problem is that some Python-dependent extensions are dependent on the Python versions used (for example the official Python extension asks for the path to the Python executable used for the current project)--how would this be resolved if Pyodide was used?

[joyceerhl]

should not cause any issues except if the final Pyodide build for NodeJS uses native NodeJS modules

FWIW it is possible to use native node modules in VS Code but this gets troublesome for extensions if the platform specific binaries have to be built and bundled with the VS Code extension. If the native node modules in question use Node-API there's no issue.

if the Pyodide instance in VSCode can be shared via an API

As a first step, we can publish an extension which exposes the Pyodide instance via an API, and then other extensions can declare a dependency on that with extensionDependencies in package.json, and access the API via https://code.visualstudio.com/api/references/vscode-api#Extension.exports

[lhl2617]

@joyceerhl thanks for that--can we discuss further in a separate issue under vscode to reduce noise here?

(VSCode-relevant discussion below)

As a first step, we can publish an extension which exposes the Pyodide instance via an API, and then other extensions can declare a dependency on that with extensionDependencies in package.json, and access the API via https://code.visualstudio.com/api/references/vscode-api#Extension.exports

I wonder if there was any discussion of making extensionDependencies an object with extension ID keys and version values? (like normal package.json dependencies).

[hoodmane]

Okay I moved the vscode discussion over to #1712 (I don't think I can actually move peoples' comments so I just quoted them over).

[hoodmane]

I am closing this as resolved. We have been running the full test suite in node as part of the CI for months now. Please feel free to open a new issue if your use case isn't supported.

[brettcannon]

This is also being tested nightly against CPython main: https://github.com/ethanhs/python-wasm/actions/workflows/ci.yml?query=branch%3Amain