Update to support Datashader on Python 3.11 #5720

Hoxbro · 2023-05-17T09:21:05Z

The reason why this failed is because of the multiprocessing of pytest-xdist and not having set multiple OMP_NUM_THREADS.

I could recreate the problem locally in a Virtual Machine on AMD CPU + Windows 10. @ianthomas23 could not recreate on an Intel CPU + Windows 11 on a real machine.

The steps I needed to recreate it was creating an environment like: conda install python=3.8 pytest nomkl pandas pyarrow pytest-xdist which yielded the following environment:

Environment

name: win_crash
channels:
  - conda-forge
dependencies:
  - aws-c-auth=0.6.27=hdef5456_1
  - aws-c-cal=0.5.26=h1b710bc_1
  - aws-c-common=0.8.19=hcfcfb64_0
  - aws-c-compression=0.2.16=h91ceee4_7
  - aws-c-event-stream=0.2.20=h3025ae5_7
  - aws-c-http=0.7.7=haa0fb2f_4
  - aws-c-io=0.13.21=hcbbf384_5
  - aws-c-mqtt=0.8.11=ha0c62a6_1
  - aws-c-s3=0.3.0=h9503c08_2
  - aws-c-sdkutils=0.1.9=h91ceee4_2
  - aws-checksums=0.1.14=h91ceee4_7
  - aws-crt-cpp=0.20.1=he8b9daf_3
  - aws-sdk-cpp=1.10.57=h353f050_12
  - bzip2=1.0.8=h8ffe710_4
  - c-ares=1.19.1=hcfcfb64_0
  - ca-certificates=2023.5.7=h56e8100_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - exceptiongroup=1.1.1=pyhd8ed1ab_0
  - execnet=1.9.0=pyhd8ed1ab_0
  - gflags=2.2.2=ha925a31_1004
  - glog=0.6.0=h4797de2_0
  - importlib-metadata=6.6.0=pyha770c72_0
  - iniconfig=2.0.0=pyhd8ed1ab_0
  - krb5=1.20.1=heb0366b_0
  - libabseil=20230125.2=cxx17_h63175ca_2
  - libarrow=12.0.0=h9bf75db_2_cpu
  - libblas=3.9.0=16_win64_openblas
  - libbrotlicommon=1.0.9=hcfcfb64_8
  - libbrotlidec=1.0.9=hcfcfb64_8
  - libbrotlienc=1.0.9=hcfcfb64_8
  - libcblas=3.9.0=16_win64_openblas
  - libcrc32c=1.1.2=h0e60522_0
  - libcurl=8.1.1=h68f0423_0
  - libevent=2.1.12=hf43717d_0
  - libffi=3.4.2=h8ffe710_5
  - libflang=5.0.0=h6538335_20180525
  - libgoogle-cloud=2.10.1=h00b2bdc_1
  - libgrpc=1.54.2=ha177ca7_2
  - liblapack=3.9.0=16_win64_openblas
  - libopenblas=0.3.21=pthreads_h02691f0_0
  - libprotobuf=3.21.12=h12be248_0
  - libsqlite=3.42.0=hcfcfb64_0
  - libssh2=1.10.0=h9a1e1f7_3
  - libthrift=0.18.1=h9ce19ad_1
  - libutf8proc=2.8.0=h82a8f57_0
  - libzlib=1.2.13=hcfcfb64_4
  - llvm-meta=5.0.0=0
  - lz4-c=1.9.4=hcfcfb64_0
  - nomkl=1.0=h5ca1d4c_0
  - numpy=1.24.3=py38h1d91fd2_0
  - openmp=5.0.0=vc14_1
  - openssl=3.1.0=hcfcfb64_3
  - orc=1.8.3=hada7b9e_0
  - packaging=23.1=pyhd8ed1ab_0
  - pandas=2.0.1=py38hf08cf0d_1
  - pip=23.1.2=pyhd8ed1ab_0
  - pluggy=1.0.0=pyhd8ed1ab_5
  - pyarrow=12.0.0=py38h75459e5_2_cpu
  - pytest=7.3.1=pyhd8ed1ab_0
  - pytest-xdist=3.3.1=pyhd8ed1ab_0
  - python=3.8.16=h4de0772_1_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-tzdata=2023.3=pyhd8ed1ab_0
  - python_abi=3.8=3_cp38
  - pytz=2023.3=pyhd8ed1ab_0
  - re2=2023.03.02=hd4eee63_0
  - setuptools=67.7.2=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.1.10=hfb803bf_0
  - tk=8.6.12=h8ffe710_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - ucrt=10.0.22621.0=h57928b3_0
  - vc=14.3=hb25d44b_16
  - vc14_runtime=14.34.31931=h5081d32_16
  - vs2015_runtime=14.34.31931=hed1258a_16
  - wheel=0.40.0=pyhd8ed1ab_0
  - xz=5.2.6=h8d14728_0
  - zipp=3.15.0=pyhd8ed1ab_0
  - zlib=1.2.13=hcfcfb64_4
  - zstd=1.5.2=h12be248_6

And having the following structure. Where conftest.py containing import pandas as pd and test_example.py was an empty file.

win_crash/
├── conftest.py
└── tests
    └── test_example.py

When running this command: pytest -n 8. Where I have 8 cores available.

The error output could, in the best case, be the following. In the worse case, it could raise a Windows error, close the command prompt or git bash, or even give freeze Windows, so I needed to restart.

Error traceback

platform win32 -- Python 3.8.16, pytest-7.3.1, pluggy-1.0.0
rootdir: C:\Users\shh\Desktop
plugins: xdist-3.3.1
collecting: 0/8 workers[gw4] node down: Traceback (most recent call last):
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 634, in _importconftest
    mod = import_path(conftestpath, mode=importmode, root=rootpath)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\pathlib.py", line 564, in import_path
    importlib.import_module(module_name)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module
    exec(co, module.__dict__)
  File "C:\Users\shh\Desktop\test\conftest.py", line 1, in <module>
    import pandas as pd
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pandas\__init__.py", line 141, in <module>
    from pandas.io.api import (
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pandas\io\api.py", line 13, in <module>
    from pandas.io.html import read_html
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 839, in exec_module
  File "<frozen importlib._bootstrap_external>", line 934, in get_code
  File "<frozen importlib._bootstrap_external>", line 1033, in get_data
MemoryError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_base.py", line 1084, in executetask
    do_exec(co, loc)  # noqa
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\remote.py", line 345, in <module>
    config = _prepareconfig(args, None)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 328, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\helpconfig.py", line 103, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 1067, in pytest_cmdline_parse
    self.parse(args)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 1354, in parse
    self._preparse(args, addopts=addopts)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 1256, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 1133, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 556, in _set_initial_conftests
    self._try_load_conftest(anchor, namespace.importmode, rootpath)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 578, in _try_load_conftest
    self._getconftestmodules(x, importmode, rootpath)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 602, in _getconftestmodules
    mod = self._importconftest(conftestpath, importmode, rootpath)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 638, in _importconftest
    raise ConftestImportFailure(conftestpath, exc_info) from e
_pytest.config.ConftestImportFailure: MemoryError:  (from C:\Users\shh\Desktop\test\conftest.py)


replacing crashed worker gw4
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\main.py", line 269, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\dsession.py", line 122, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\dsession.py", line 145, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\dsession.py", line 234, in worker_errordown
INTERNALERROR>     self._clone_node(node)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\dsession.py", line 339, in _clone_node
INTERNALERROR>     node = self.nodemanager.setup_node(spec, self.queue.put)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\workermanage.py", line 71, in setup_node
INTERNALERROR>     gw = self.group.makegateway(spec)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\multi.py", line 133, in makegateway
INTERNALERROR>     io = gateway_io.create_io(spec, execmodel=self.execmodel)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_io.py", line 118, in create_io
INTERNALERROR>     return Popen2IOMaster(args, execmodel)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_io.py", line 21, in __init__
INTERNALERROR>     self.popen = p = execmodel.PopenPiped(args)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_base.py", line 184, in PopenPiped
INTERNALERROR>     return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE)
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\subprocess.py", line 858, in __init__
INTERNALERROR>     self._execute_child(args, executable, preexec_fn, close_fds,
INTERNALERROR>   File "C:\Users\shh\miniconda3\envs\win_crash\lib\subprocess.py", line 1311, in _execute_child
INTERNALERROR>     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
INTERNALERROR> OSError: [WinError 1450] Insufficient system resources exist to complete the requested service
Traceback (most recent call last):
  File "C:\Users\shh\miniconda3\envs\win_crash\Scripts\pytest-script.py", line 9, in <module>
    sys.exit(console_main())
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 189, in console_main
    code = main()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\config\__init__.py", line 166, in main
    ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\main.py", line 316, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\main.py", line 304, in wrap_session
    config.hook.pytest_sessionfinish(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\_pytest\terminal.py", line 812, in pytest_sessionfinish
    outcome.get_result()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\pluggy\_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\dsession.py", line 92, in pytest_sessionfinish
    nm.teardown_nodes()
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\xdist\workermanage.py", line 81, in teardown_nodes
    self.group.terminate(self.EXIT_TIMEOUT)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\multi.py", line 215, in terminate
    safe_terminate(
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\multi.py", line 308, in safe_terminate
    reply = workerpool.spawn(termkill, termfunc, killfunc)
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_base.py", line 315, in spawn
    self.execmodel.start(self._perform_spawn, (reply,))
  File "C:\Users\shh\miniconda3\envs\win_crash\lib\site-packages\execnet\gateway_base.py", line 93, in exec_start
    self._start_new_thread(func, args)
RuntimeError: can't start new thread

Example of crash:

With small changes to the set-up the code would run just fine:

Running pytest -n 7 - Though it would fail if I had VS Code open and ran the command in Command prompt...
Commenting out import pandas as pd in the conftest.py
Having set the environment variable OMP_NUM_THREADS=1

codecov-commenter · 2023-05-17T15:52:10Z

Codecov Report

Merging #5720 (ebbad8a) into main (f4c4c12) will decrease coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #5720      +/-   ##
==========================================
- Coverage   88.24%   88.19%   -0.05%     
==========================================
  Files         304      304              
  Lines       62680    62680              
==========================================
- Hits        55309    55283      -26     
- Misses       7371     7397      +26

see 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Hoxbro added 3 commits May 17, 2023 11:19

Update to support Numba on Python 3.11

663a89b

Reduce test matrix

8550d93

Debug mode

359bd88

Hoxbro force-pushed the update_numba branch from bd5df1a to 359bd88 Compare May 17, 2023 10:49

Add bokeh 2

5e97762

Hoxbro added 10 commits May 17, 2023 18:09

Make pytest multi back

fbe5e3e

Set NUMBA_NUM_THREADS

d4b03ac

Add nomkl

5d81626

Remove numba channel

05cf8d4

Try something

3e09a9e

Update holoviz task

bfb6d8f

Upper pin numpy in tests

d9aca8a

Remove pandas from conftest

bc6be1a

Try OPM_NUM_THREADS

734d210

Clean up

55f6221

Hoxbro force-pushed the update_numba branch from b6b12d3 to 55f6221 Compare May 24, 2023 16:59

Hoxbro added 4 commits May 24, 2023 20:27

Add OPM_NUM_THREADS back

44a025d

Update comment in conftest.py

ecf4706

Update test matrix with other os

ac5f606

Update text

ebbad8a

Hoxbro merged commit 2afd868 into main Jun 2, 2023
13 checks passed

Hoxbro deleted the update_numba branch June 2, 2023 10:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to support Datashader on Python 3.11 #5720

Update to support Datashader on Python 3.11 #5720

Hoxbro commented May 17, 2023 •

edited

codecov-commenter commented May 17, 2023 •

edited

Update to support Datashader on Python 3.11 #5720

Update to support Datashader on Python 3.11 #5720

Conversation

Hoxbro commented May 17, 2023 • edited

codecov-commenter commented May 17, 2023 • edited

Codecov Report

Hoxbro commented May 17, 2023 •

edited

codecov-commenter commented May 17, 2023 •

edited