multithreading example dumps a stacktrace all while running to completion

[teh-cmc]

$ ./examples/python/multithreading/main.py
Traceback (most recent call last):
  File "pyarrow/pandas-shim.pxi", line 110, in pyarrow.lib._PandasAPIShim._check_import
  File "pyarrow/pandas-shim.pxi", line 48, in pyarrow.lib._PandasAPIShim._import_pandas
  File "/home/cmc/dev/rerun-io/rerun/venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 24, in <module>
    import concurrent.futures.thread  # noqa
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/lib/python3.10/threading.py", line 1504, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown
Exception ignored in: 'pyarrow.lib._PandasAPIShim._have_pandas_internal'
Traceback (most recent call last):
  File "pyarrow/pandas-shim.pxi", line 110, in pyarrow.lib._PandasAPIShim._check_import
  File "pyarrow/pandas-shim.pxi", line 48, in pyarrow.lib._PandasAPIShim._import_pandas
  File "/home/cmc/dev/rerun-io/rerun/venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 24, in <module>
    import concurrent.futures.thread  # noqa
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 37, in <module>
    threading._register_atexit(_python_exit)
  File "/usr/lib/python3.10/threading.py", line 1504, in _register_atexit
    raise RuntimeError("can't register atexit after shutdown")
RuntimeError: can't register atexit after shutdown

$ echo $?
0

[jleibs]

Does this happen deterministically for you? I can't repro locally.

[jleibs]

Oh... this only happens if pandas is installed...

[jleibs]

Ok, a bit more digging on this... this only happens if (1) pandas is installed, and (2) we use pyarrow from inside of a thread.

Minimal repro without touching rerun:

import threading
import pyarrow


def use_pyarrow() -> None:
    table = pyarrow.table({"a": [1, 2, 3]})


def main() -> None:
    t = threading.Thread(target=use_pyarrow, args=())
    t.start()


if __name__ == "__main__":
    main()

Will create an issue on pyarrow github.

[jleibs]

Upstream problem for reference:
apache/arrow#35237