Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multithreading example dumps a stacktrace all while running to completion #1932

Closed
teh-cmc opened this issue Apr 19, 2023 · 4 comments · Fixed by #1934
Closed

multithreading example dumps a stacktrace all while running to completion #1932

teh-cmc opened this issue Apr 19, 2023 · 4 comments · Fixed by #1934
Labels
🪳 bug Something isn't working 🧑‍💻 dev experience developer experience (excluding CI) examples Issues relating to the Rerun examples

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Apr 19, 2023

$ ./examples/python/multithreading/main.py
Traceback (most recent call last):
  File "pyarrow/pandas-shim.pxi", line 110, in pyarrow.lib._PandasAPIShim._check_import
  File "pyarrow/pandas-shim.pxi", line 48, in pyarrow.lib._PandasAPIShim._import_pandas
  File "/home/cmc/dev/rerun-io/rerun/venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 24, in <module>
    import concurrent.futures.thread  # noqa
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/lib/python3.10/threading.py", line 1504, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown
Exception ignored in: 'pyarrow.lib._PandasAPIShim._have_pandas_internal'
Traceback (most recent call last):
  File "pyarrow/pandas-shim.pxi", line 110, in pyarrow.lib._PandasAPIShim._check_import
  File "pyarrow/pandas-shim.pxi", line 48, in pyarrow.lib._PandasAPIShim._import_pandas
  File "/home/cmc/dev/rerun-io/rerun/venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 24, in <module>
    import concurrent.futures.thread  # noqa
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/lib/python3.10/threading.py", line 1504, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

$ echo $?
0
@teh-cmc teh-cmc added 🪳 bug Something isn't working 🧑‍💻 dev experience developer experience (excluding CI) examples Issues relating to the Rerun examples labels Apr 19, 2023
@teh-cmc teh-cmc mentioned this issue Apr 19, 2023
@jleibs
Copy link
Member

jleibs commented Apr 19, 2023

Does this happen deterministically for you? I can't repro locally.

@jleibs
Copy link
Member

jleibs commented Apr 19, 2023

Oh... this only happens if pandas is installed...

@jleibs
Copy link
Member

jleibs commented Apr 19, 2023

Ok, a bit more digging on this... this only happens if (1) pandas is installed, and (2) we use pyarrow from inside of a thread.

Minimal repro without touching rerun:

import threading
import pyarrow


def use_pyarrow() -> None:
    table = pyarrow.table({"a": [1, 2, 3]})


def main() -> None:
    t = threading.Thread(target=use_pyarrow, args=())
    t.start()


if __name__ == "__main__":
    main()

Will create an issue on pyarrow github.

@jleibs
Copy link
Member

jleibs commented Apr 19, 2023

Upstream problem for reference:
apache/arrow#35237

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🪳 bug Something isn't working 🧑‍💻 dev experience developer experience (excluding CI) examples Issues relating to the Rerun examples
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants