windows python3.9 CI builds failing to import torch: Error loading "...\caffe2_detectron_ops_gpu.dll" or one of its dependencies

[graingert]

2022-02-04T10:40:01.6126334Z =================================== ERRORS ====================================
2022-02-04T10:40:01.6126653Z �[31m�[1m__________ ERROR collecting distributed/protocol/tests/test_torch.py __________�[0m
2022-02-04T10:40:01.6127029Z �[1m�[31mdistributed\protocol\tests\test_torch.py�[0m:6: in <module>
2022-02-04T10:40:01.6127325Z     torch = pytest.importorskip("torch")
2022-02-04T10:40:01.6127710Z �[1m�[31mC:\Miniconda3\envs\dask-distributed\lib\site-packages\torch\__init__.py�[0m:124: in <module>
2022-02-04T10:40:01.6129794Z     raise err
2022-02-04T10:40:01.6130366Z �[1m�[31mE   OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\Miniconda3\envs\dask-distributed\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.�[0m
2022-02-04T10:40:01.6130942Z ------------------------------- Captured stderr -------------------------------
2022-02-04T10:40:01.6131295Z Windows fatal exception: code 0xc0000138
2022-02-04T10:40:01.6131461Z 
2022-02-04T10:40:01.6131583Z Thread 0x000016e8 (most recent call first):
2022-02-04T10:40:01.6131982Z   File Windows fatal exception: "code 0xC:\Miniconda3\envs\dask-distributed\lib\concurrent\futures\thread.pyc0000138"
2022-02-04T10:40:01.6132241Z 
2022-02-04T10:40:01.6132325Z , line 81 in _worker
2022-02-04T10:40:01.6132637Z   File "C:\Miniconda3\envs\dask-distributed\lib\threading.py", line 910 in run
2022-02-04T10:40:01.6133038Z   File "C:\Miniconda3\envs\dask-distributed\lib\threading.py", line 973 in _bootstrap_inner
2022-02-04T10:40:01.6133469Z   File "C:\Miniconda3\envs\dask-distributed\lib\threading.py", line 930 in _bootstrap
2022-02-04T10:40:01.6133698Z 
2022-02-04T10:40:01.6133816Z Thread 0x000008b4 (most recent call first):
2022-02-04T10:40:01.6134172Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\torch\__init__.py", line 119 in <module>
2022-02-04T10:40:01.6134596Z   File "<frozen importlib._bootstrap>", line 228 in _call_with_frames_removed
2022-02-04T10:40:01.6134927Z   File "<frozen importlib._bootstrap_external>", line 850 in exec_module
2022-02-04T10:40:01.6135269Z   File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
2022-02-04T10:40:01.6135601Z   File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
2022-02-04T10:40:01.6135915Z   File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
2022-02-04T10:40:01.6136331Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\outcomes.py", line 208 in importorskip
2022-02-04T10:40:01.6136814Z   File "D:\a\distributed\distributed\distributed\protocol\tests\test_torch.py", line 6 in <module>
2022-02-04T10:40:01.6137283Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\assertion\rewrite.py", line 170 in exec_module
2022-02-04T10:40:01.6137714Z   File "<frozen importlib._bootstrap>", line 680 in _load_unlocked
2022-02-04T10:40:01.6138159Z   File "<frozen importlib._bootstrap>", line 986 in _find_and_load_unlocked
2022-02-04T10:40:01.6138498Z   File "<frozen importlib._bootstrap>", line 1007 in _find_and_load
2022-02-04T10:40:01.6138814Z   File "<frozen importlib._bootstrap>", line 1030 in _gcd_import
2022-02-04T10:40:01.6139226Z   File "C:\Miniconda3\envs\dask-distributed\lib\importlib\__init__.py", line 127 in import_module
2022-02-04T10:40:01.6139726Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\pathlib.py", line 524 in import_path
2022-02-04T10:40:01.6140212Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\python.py", line 578 in _importtestmodule
2022-02-04T10:40:01.6140692Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\python.py", line 500 in _getobj
2022-02-04T10:40:01.6141155Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\python.py", line 291 in obj
2022-02-04T10:40:01.6141633Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\python.py", line 516 in _inject_setup_module_fixture
2022-02-04T10:40:01.6142128Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\python.py", line 503 in collect
2022-02-04T10:40:01.6142566Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\runner.py", line 341 in <lambda>
2022-02-04T10:40:01.6143027Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\runner.py", line 311 in from_call
2022-02-04T10:40:01.6143518Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\runner.py", line 341 in pytest_make_collect_report
2022-02-04T10:40:01.6144347Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
2022-02-04T10:40:01.6144835Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
2022-02-04T10:40:01.6145293Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
2022-02-04T10:40:01.6145763Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\runner.py", line 458 in collect_one_node
2022-02-04T10:40:01.6146234Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 808 in genitems
2022-02-04T10:40:01.6146685Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 634 in perform_collect
2022-02-04T10:40:01.6147178Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 333 in pytest_collection
2022-02-04T10:40:01.6147669Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
2022-02-04T10:40:01.6148119Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
2022-02-04T10:40:01.6148573Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
2022-02-04T10:40:01.6149018Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 322 in _main
2022-02-04T10:40:01.6149465Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 269 in wrap_session
2022-02-04T10:40:01.6149986Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\main.py", line 316 in pytest_cmdline_main
2022-02-04T10:40:01.6150447Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
2022-02-04T10:40:01.6150907Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
2022-02-04T10:40:01.6151365Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
2022-02-04T10:40:01.6151811Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\config\__init__.py", line 162 in main
2022-02-04T10:40:01.6152282Z   File "C:\Miniconda3\envs\dask-distributed\lib\site-packages\_pytest\config\__init__.py", line 185 in console_main
2022-02-04T10:40:01.6152808Z   File "C:\Miniconda3\envs\dask-distributed\Scripts\pytest-script.py", line 10 in <module>
2022-02-04T10:40:01.6153213Z ----- generated xml file: D:\a\distributed\distributed\reports\pytest.xml -----
2022-02-04T10:40:01.6153447Z 
2022-02-04T10:40:01.6153613Z ---------- coverage: platform win32, python 3.9.10-final-0 -----------
2022-02-04T10:40:01.6153912Z Coverage XML written to file coverage.xml
2022-02-04T10:40:01.6154067Z 
2022-02-04T10:40:01.6154193Z =========================== short test summary info ===========================
2022-02-04T10:40:01.6154549Z SKIPPED [1] distributed\comm\tests\test_ucx.py:9: could not import 'ucp': No module named 'ucp'
2022-02-04T10:40:01.6154971Z SKIPPED [1] distributed\comm\tests\test_ucx_config.py:20: could not import 'ucp': No module named 'ucp'
2022-02-04T10:40:01.6155446Z SKIPPED [1] distributed\diagnostics\tests\test_widgets.py:3: could not import 'ipywidgets': No module named 'ipywidgets'
2022-02-04T10:40:01.6155934Z SKIPPED [1] distributed\protocol\tests\test_arrow.py:3: could not import 'pyarrow': No module named 'pyarrow'
2022-02-04T10:40:01.6156376Z SKIPPED [1] distributed\protocol\tests\test_cupy.py:9: could not import 'cupy': No module named 'cupy'
2022-02-04T10:40:01.6156835Z SKIPPED [1] distributed\protocol\tests\test_keras.py:3: could not import 'keras': No module named 'tensorflow'
2022-02-04T10:40:01.6157294Z SKIPPED [1] distributed\protocol\tests\test_numba.py:9: could not import 'numba.cuda': No module named 'numba'
2022-02-04T10:40:01.6157770Z SKIPPED [1] distributed\protocol\tests\test_rmm.py:8: could not import 'numba.cuda': No module named 'numba'
2022-02-04T10:40:01.6158228Z SKIPPED [1] distributed\protocol\tests\test_sparse.py:4: could not import 'sparse': No module named 'sparse'
2022-02-04T10:40:01.6158621Z ERROR distributed/protocol/tests/test_torch.py - OSError: [WinError 182] The ...
2022-02-04T10:40:01.6158967Z !!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

[graingert]

most recent build: https://github.com/dask/distributed/runs/5065082062?check_suite_focus=true#step:12:48
first failing build: https://github.com/dask/distributed/runs/5059436423?check_suite_focus=true
last passing build: https://github.com/dask/distributed/runs/5052697667?check_suite_focus=true

[graingert]

@@ -54,6 +54,7 @@
    - filelock=3.4.2=pyhd8ed1ab_1
    - filesystem-spec=0.2.0=py_0
    - freetype=2.10.4=h546665d_1
+   - fribidi=1.0.10=h8d14728_0
    - frozenlist=1.3.0=py39hb82d6ee_0
    - fsspec=2022.1.0=pyhd8ed1ab_0
    - h5py=3.6.0=nompi_py39hd4deaf1_100
@@ -78,6 +79,7 @@
    - libcurl=7.81.0=h789b8ee_0
    - libdeflate=1.8=h8ffe710_0
    - libffi=3.4.2=h8ffe710_5
+   - libimagequant=2.17.0=hcfcfb64_1
    - liblapack=3.9.0=13_win64_mkl
    - liblapacke=3.9.0=13_win64_mkl
    - libnetcdf=4.8.1=nompi_h1cc8e9d_101
@@ -86,8 +88,12 @@
    - libssh2=1.10.0=h9a1e1f7_2
    - libtiff=4.3.0=hd413186_2
    - libuv=1.43.0=h8ffe710_0
+   - libwebp=1.2.2=h57928b3_0
+   - libwebp-base=1.2.2=h8ffe710_1
+   - libxcb=1.13=hcd874cb_1004
    - libzip=1.8.0=h519de47_1
    - libzlib=1.2.11=h8ffe710_1013
+   - llvm-openmp=12.0.1=h2d74725_1
    - locket=0.2.0=py_2
    - lz4=3.1.10=py39h92e281b_0
    - lz4-c=1.9.3=h8ffe710_1
@@ -107,20 +113,20 @@
    - netcdf4=1.5.8=nompi_py39hf113b1f_101
    - nodeenv=1.6.0=pyhd8ed1ab_0
    - numpy=1.22.1=py39h6331f09_0
-   - olefile=0.46=pyh9f0ad1d_1
    - openjpeg=2.4.0=hb211442_1
    - openssl=3.0.0=h8ffe710_2
    - packaging=21.3=pyhd8ed1ab_0
    - pandas=1.4.0=py39h2e25243_0
    - paramiko=2.9.2=pyhd8ed1ab_0
    - partd=1.2.0=pyhd8ed1ab_0
-   - pillow=8.4.0=py39h916092e_0
+   - pillow=9.0.0=py39ha53f419_0
    - pip=22.0.3=pyhd8ed1ab_0
    - platformdirs=2.3.0=pyhd8ed1ab_0
    - pluggy=1.0.0=py39hcbf5309_2
    - pre-commit=2.17.0=py39hcbf5309_0
    - prometheus_client=0.13.1=pyhd8ed1ab_0
    - psutil=5.9.0=py39hb82d6ee_0
+   - pthread-stubs=0.4=hcd874cb_1001
    - py=1.11.0=pyh6c4a22f_0
    - pycparser=2.21=pyhd8ed1ab_0
    - pynacl=1.5.0=py39hb3671d1_0
@@ -178,6 +184,8 @@
    - wheel=0.37.1=pyhd8ed1ab_0
    - win_inet_pton=1.1.0=py39hcbf5309_3
    - wrapt=1.13.3=py39hb82d6ee_1
+   - xorg-libxau=1.0.9=hcd874cb_0
+   - xorg-libxdmcp=1.1.3=hcd874cb_0
    - xz=5.2.5=h62dcd97_1
    - yaml=0.2.5=h8ffe710_2
    - yarl=1.7.2=py39hb82d6ee_1
@@ -187,7 +195,7 @@
    - zstandard=0.17.0=py39hb82d6ee_0
    - zstd=1.5.2=h6255e5f_0
    - pip:
-     - dask==2022.1.1+12.g5e8a4813
+     - dask==2022.1.1+13.g6824e5dd
      - joblib==1.2.0.dev0
      - keras==2.8.0
      - s3fs==2022.1.0+5.g736ee5a

[graingert]

pytorch/pytorch#72293
conda-forge/conda-forge.github.io#1597

[jrbourbeau]

Re-opening to track when we can undo the pinning added in #5755