-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GH-41910: [Python] Add support for Pyodide #37822
base: main
Are you sure you want to change the base?
Conversation
@kou And these are the python changes |
@joemarshall thanks for the PR! We might want to expose
This part is not actually included here? (or I don't understand the sentence) |
Sorry, I missed out putting in the setup.py changes. They're in now. About is_threading_enabled(), it is currently in |
Oh and for now I have blocked the auto-setting of PYARROW_* to happen only on emscripten - I don't know if that makes sense or not, but it isn't possible to build for emscripten without that change or something similar right now. |
@jorisvandenbossche That's a weird error - I think it must have not loaded pyodide in node that time for some reason. Looking at the output I think it may be because on the CI system buffering of the pipe between node and python runner is happening differently and it is losing some data to stdin. I've pushed a fix which should hopefully make this not happen. |
@github-actions crossbow submit test-conda-python-emscripten |
|
@github-actions crossbow submit test-conda-python-emscripten |
Revision: 1a19b36 Submitted crossbow builds: ursacomputing/crossbow @ actions-73fe2c1343
|
|
||
# Install basic build stuff, don't pin versions, hence ignore lint | ||
# hadolint ignore=DL3008 | ||
RUN apt-get update && apt-get install --no-install-recommends -y -q unzip zip libpthread-stubs0-dev build-essential && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use conda to install them?
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
Co-authored-by: Sutou Kouhei <kou@cozmixng.org>
@kou I've updated based on those comments - it now gets emscripten version from pyodide, and uses conda to install nodejs etc. I also removed the use of multiprocessing in the python test runner, to see if that is causing the weird hang in node tests, which I just can't reproduce in docker here even using archery. If you could run tests again that would be great. |
@github-actions crossbow submit test-conda-python-emscripten |
Revision: e2d8d73 Submitted crossbow builds: ursacomputing/crossbow @ actions-a69d7a8e48
|
Aha, it doesn't work if there isn't a tty. I think I've fixed the node.js tests now (at least they work with nohup now) |
@github-actions crossbow submit test-conda-python-emscripten |
Revision: 46e179a Submitted crossbow builds: ursacomputing/crossbow @ actions-255fa63cba
|
@joemarshall every time the build fails... Keep it up it's soooo close! You probably deserve another huge cash dump for all the work done here. |
FYI: We can run the job on local: |
pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.
Split from #37696