Upgrade to Pyodide 0.23.4 (with Python 3.11) and compile dependencies

[alexmojaki]

No description provided.

[paulfitz]

(@alexmojaki can you rebase this when you have a moment)

[paulfitz]

From sandbox/pyodide, I ran build_packages.sh and it ran to completion, building everything needed 👍. I then tried running GRIST_SANDBOX_FLAVOR=pyodide yarn start and opening a document, but I got this error:

2023-07-31 18:03:50.301 - info: Sandbox stderr: [pyodide sandbox] [package] Loaded et_xmlfile, executing, asttokens, iso8601, lazy_object_proxy, chardet, pure_eval, python_dateutil, astroid, sortedcontainers, six, unittest_xml_reporting, stack_data, roman, wrapt, typing_extensions, openpyxl, friendly_traceback, phonenumberslite sandboxPid=692865, flavor=pyodide, command=undefined, entryPoint=(default), docId=smStzDu4B6G5AT4CmVxdua
2023-07-31 18:03:50.394 - info: Sandbox stderr: [pyodide error] PythonError: Traceback (most recent call last):
  File "/lib/python3.10/_pyodide/_base.py", line 460, in eval_code
    .run(globals, locals)
  File "/lib/python3.10/_pyodide/_base.py", line 306, in run
    coroutine = eval(self.code, globals, locals)
  File "<exec>", line 5, in <module>
  File "/grist/grist.py", line 8, in <module>
    from usertypes import Any, Text, Blob, Int, Bool, Date, DateTime, \
  File "/grist/usertypes.py", line 21, in <module>
    import six
ImportError: bad magic number in 'six': b'\xa7\r\r

I saw a pyodide version number in setup.sh that looked out of step with the rest of the PR, updating that and replacing _build/worker seemed to do the trick.

For the unrelated test failure - I made a series of changes to test/nbrowser/SelectBySummary.ts that could be worth considering for test/nbrowser/SelectByRefList.ts - can't actually say if any of them change anything, since I can't duplicate the failure, so I'm just whittling away at possibilities. The SelectBy tests have been flakey since the order of test file evaluation became less deterministic.

I'd like to run make save_packages to save the painstakingly built output, but I wonder if there needs to be some kind of overall version number in the path? Are any build files that happen to have the same name currently usable by grist-static in its current state?

[paulfitz]

I'd like to run make save_packages to save the painstakingly built output, but I wonder if there needs to be some kind of overall version number in the path? Are any build files that happen to have the same name currently usable by grist-static in its current state?

Just for clarity on this - pyodide is the default python engine for grist-electron these days. As a desktop app grist-electron gets build on a matrix of platforms, so building the wheels for it repeatedly would be a significant slowdown. So it is best if the wheels are cached. See e.g.:
https://github.com/gristlabs/grist-electron/blob/ee6e73cfe8a8464d84538c826ab3edcaa03f3c71/scripts/setup.sh#L170-L176

I think when the pyodide engine in grist-core is updated, we should probably assign it a version and cache material in a distinct path. Then grist-electron and grist-static can get updated on their own schedule.

[alexmojaki]

I didn't realise the pyodide sandbox flavor was actually still being used!

I think grouping the packages by pyodide version makes sense.

[paulfitz]

I didn't realise the pyodide sandbox flavor was actually still being used!

It is a relatively new thing. I cobbled together the pyodide sandbox in order to have some kind of sandboxing option for Windows users of the desktop app. Then realized that it came close to making something like grist-static possible, so I started hacking on that. None of this was really planned, it was a rogue project.

I think grouping the packages by pyodide version makes sense.

Who should make that happen? Do you plan to do it in this PR? This PR works well for me now, the one remaining issue is the package layout. Currently if you follow the README in sandbox/pyodide, you'll end up getting packages for a different version from https://s3.amazonaws.com/grist-pynbox/pyodide/packages/

[paulfitz]

(The sporadically failing TypeChange.ntest test should be fixed on main now)

[alexmojaki]

Rather than the pyodide version, let's just use a custom version number for this purpose to also account for things like changes in requirements. So e.g. we could start with s3://grist-pynbox/pyodide/packages/v2/ and then later change that to v3 etc.

Let's try to avoid accidentally deleting/replacing uploaded files if we forget to update the version number. Is there an easy alternative to aws s3 sync that fails if there's already files in that 'directory'? Otherwise, a low tech solution could be to remove save_packages from Makefile and explain what to do in the README so that people are more likely to do it carefully.

If you could take over from here I'd really appreciate that, bit overwhelmed atm

[paulfitz]

Let's land this as is, I'll follow up quickly with something that adds versioning. Thanks @alexmojaki.

[paulfitz]

Rather than the pyodide version, let's just use a custom version number for this purpose to also account for things like changes in requirements. So e.g. we could start with s3://grist-pynbox/pyodide/packages/v2/ and then later change that to v3 etc.

Sounds good.

Let's try to avoid accidentally deleting/replacing uploaded files if we forget to update the version number. Is there an easy alternative to aws s3 sync that fails if there's already files in that 'directory'? Otherwise, a low tech solution could be to remove save_packages from Makefile and explain what to do in the README so that people are more likely to do it carefully.

There may be a flag, I can check. The set of people who can upload to grist-pynbox is very low though.