Build libraries in a seperate repository

[ryanking13]

Unlike Python packages, static and shared libraries included in Pyodide repository are built with naive emcc commands, not our pywasmcross wrapper. So it can be easily built from separate repositories.

So I think it would be nice if we can prebuild some of the libraries (which doesn't have complicated build scripts) in another repo and use the built libraries in our CI. Probably we can make a new organization like pyodide-ports and create repositories in it.

For example, in meta.yaml:

# libxml2.zip containing `libxml2.so` and headers.
# Then, we can just uncompress it to `WASM_LIBRARY_DIR` during the build process
source:
    url: http://.../libxml2.zip
    prebuilt: true

Advantages

1. Can reduce CI time dramatically.
2. People can easily re-use them when building their own python package. For now, we only vendor some of it in our xbuildenv, so it is a bit hard to use these libraries externally.

Disadvantages

1. Adds maintenance burden
2. Updating the Emscripten version will become a bit more stressful. (Though maybe we can automate some of it...)
3. Updating flags for side modules will become a bit more stressful.

Partially related to @joemarshall's issue #3336

WDYT?

[rth]

Well the ideal case IMO would be to build all packages in a separate repo. Libraries indeed don't need pywasmcross but they are still tied to a specific emscripten version used in Pyodide and the associated flags. So in my opinion having a separate setup just for libraries, wouldn't really address our scalability issues, while making how packages are built less homogeneous.

Ideally the workflow I would like is to have,

• a repo with only meta.yaml and corresponding tests that builds for a given Pyodide stable release.
• for a new PR build only the changed / added packages
• deploy them to the CDN under some path that includes the hash of the emscripten/pyodide version
• have a way to create a distribution repodata.json from either these previously built packages. This would also then make it more realistic to optionally run the test suite of packages.
• we could still build & test a subset of packages we want in the main repo.

It's still a bit blocked by #3049
Yeah, overall it's potentially more maintenance, but I also don't see how we can go on with the current system with people requesting more packages, us unwilling to build them as part of the CI (particularly for very long running buids) and even if someone already built the corresponding wheel not having a way to discover that a wheel was already built by someone for emscripten/wasm32