test_distribution in test_datasets.py not running on CircleCI

[k-dominik]

The test test_distribution from tests/test_datasets.py is timing out on CircleCI. When starting the test with ssh enabled on CircleCI the test passes. I also added some print statements in the test and then the test would run through. with faulthandler_timeout I got the following traceback:

Thread 0x00007fb4aa15a740 (most recent call first):
  File "/root/repo/tiktorch/server/datasets.py", line 140 in __iter__
  File "/root/repo/tests/test_server/test_datasets.py", line 138 in <genexpr>
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/collections/__init__.py", line 679 in update
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/collections/__init__.py", line 593 in __init__
  File "/root/repo/tests/test_server/test_datasets.py", line 138 in test_distribution
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/python.py", line 1641 in runtest
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 311 in from_call
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/main.py", line 323 in _main
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/site-packages/pytest/__main__.py", line 5 in <module>
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/runpy.py", line 87 in _run_code
  File "/opt/conda/envs/tiktorch-server-env/lib/python3.9/runpy.py", line 197 in _run_module_as_main

The environment on CI (linux, same as on my local machine):

channels:
  - ilastik-forge/label/freepy
  - pytorch
  - ilastik-forge
  - conda-forge
  - local
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_llvm
  - anaconda-client=1.8.0=pyhd8ed1ab_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - attrs=21.2.0=pyhd8ed1ab_0
  - beautifulsoup4=4.10.0=pyha770c72_0
  - black=21.11b0=pyhd8ed1ab_0
  - blas=2.112=mkl
  - blas-devel=3.9.0=12_linux64_mkl
  - blosc=1.21.0=h9c3ff4c_0
  - brotli=1.0.9=h7f98852_6
  - brotli-bin=1.0.9=h7f98852_6
  - brotlipy=0.7.0=py39h3811e60_1003
  - brunsli=0.1=h9c3ff4c_0
  - bump2version=1.0.1=pyh9f0ad1d_0
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - c-blosc2=2.0.4=h5f21a17_1
  - ca-certificates=2021.10.8=ha878542_0
  - cached-property=1.5.2=hd8ed1ab_1
  - cached_property=1.5.2=pyha770c72_1
  - certifi=2021.10.8=py39hf3d152e_1
  - cffi=1.15.0=py39h4bc2ebd_0
  - cfgv=3.3.1=pyhd8ed1ab_0
  - cfitsio=4.0.0=h9a35b8e_0
  - chardet=4.0.0=py39hf3d152e_2
  - charls=2.2.0=h9c3ff4c_0
  - charset-normalizer=2.0.0=pyhd8ed1ab_0
  - click=8.0.3=py39hf3d152e_1
  - cloudpickle=2.0.0=pyhd8ed1ab_0
  - clyent=1.2.2=py_1
  - colorama=0.4.4=pyh9f0ad1d_0
  - conda=4.10.3=py39hf3d152e_4
  - conda-build=3.21.6=py39hf3d152e_1
  - conda-package-handling=1.7.3=py39h3811e60_1
  - cpuonly=2.0=0
  - cryptography=36.0.0=py39h95dcef6_0
  - cudatoolkit=11.1.1=h6406543_9
  - cudnn=8.2.1.32=h86fa8c9_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - cytoolz=0.11.2=py39h3811e60_1
  - dask-core=2021.11.2=pyhd8ed1ab_0
  - dataclasses=0.8=pyhc8e2a94_3
  - dill=0.3.4=pyhd8ed1ab_0
  - distlib=0.3.3=pyhd8ed1ab_0
  - editdistance-s=1.0.0=py39h1a9c180_2
  - ffmpeg=4.3=hf484d3e_0
  - filelock=3.4.0=pyhd8ed1ab_0
  - flake8=4.0.1=pyhd8ed1ab_1
  - fonttools=4.28.1=py39h3811e60_0
  - freetype=2.10.4=h0708190_1
  - fsspec=2021.11.0=pyhd8ed1ab_0
  - giflib=5.2.1=h36c2ea0_2
  - glob2=0.7=py_0
  - gmp=6.2.1=h58526e2_0
  - gnutls=3.6.13=h85f3911_1
  - grpcio=1.33.2=py39h0c26d4f_2
  - grpcio-tools=1.33.2=py39h58526e2_0
  - h5py=3.1.0=nompi_py39h25020de_100
  - hdf5=1.10.6=nompi_h6a2412b_1114
  - icu=69.1=h9c3ff4c_0
  - identify=2.3.7=pyhd8ed1ab_0
  - idna=3.1=pyhd3deb0d_0
  - imagecodecs=2021.11.20=py39h571908b_0
  - imageio=2.9.0=py_0
  - importlib-metadata=4.2.0=py39hf3d152e_0
  - importlib_metadata=4.2.0=hd8ed1ab_0
  - importlib_resources=5.4.0=pyhd8ed1ab_0
  - inferno=v0.4.2=py_1
  - iniconfig=1.1.1=pyh9f0ad1d_0
  - ipython_genutils=0.2.0=py_1
  - isort=5.10.1=pyhd8ed1ab_0
  - jbig=2.1=h7f98852_2003
  - jinja2=3.0.3=pyhd8ed1ab_0
  - joblib=1.1.0=pyhd8ed1ab_0
  - jpeg=9d=h36c2ea0_0
  - jsonschema=4.2.1=pyhd8ed1ab_0
  - jupyter_core=4.9.1=py39hf3d152e_1
  - jxrlib=1.1=h7f98852_2
  - kiwisolver=1.3.2=py39h1a9c180_1
  - krb5=1.19.2=hcc1bbae_3
  - lame=3.100=h7f98852_1001
  - lapack=3.9.0=netlib
  - lcms2=2.12=hddcbb42_0
  - ld_impl_linux-64=2.36.1=hea4e1c9_2
  - lerc=3.0=h9c3ff4c_0
  - libaec=1.0.6=h9c3ff4c_0
  - libarchive=3.5.2=hccf745f_1
  - libblas=3.9.0=12_linux64_mkl
  - libbrotlicommon=1.0.9=h7f98852_6
  - libbrotlidec=1.0.9=h7f98852_6
  - libbrotlienc=1.0.9=h7f98852_6
  - libcblas=3.9.0=12_linux64_mkl
  - libcurl=7.80.0=h2574ce0_0
  - libdeflate=1.8=h7f98852_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=11.2.0=h1d223b6_11
  - libgfortran-ng=11.2.0=h69a702a_11
  - libgfortran5=11.2.0=h5c6108e_11
  - libiconv=1.16=h516909a_0
  - liblapack=3.9.0=12_linux64_mkl
  - liblapacke=3.9.0=12_linux64_mkl
  - liblief=0.11.5=h9c3ff4c_1
  - libnghttp2=1.43.0=h812cca2_1
  - libpng=1.6.37=h21135ba_2
  - libprotobuf=3.19.1=h780b84a_0
  - libssh2=1.10.0=ha56f1ee_2
  - libstdcxx-ng=11.2.0=he4da1e4_11
  - libtiff=4.3.0=h6f004c6_2
  - libuv=1.42.0=h7f98852_0
  - libwebp-base=1.2.1=h7f98852_0
  - libxml2=2.9.12=h885dcf4_1
  - libzlib=1.2.11=h36c2ea0_1013
  - libzopfli=1.0.3=h9c3ff4c_0
  - llvm-openmp=12.0.1=h4bd325d_1
  - locket=0.2.0=py_2
  - lz4-c=1.9.3=h9c3ff4c_1
  - lzo=2.10=h516909a_1000
  - markupsafe=2.0.1=py39h3811e60_1
  - marshmallow=3.12.2=pyhd8ed1ab_0
  - marshmallow-union=0.1.15.post1=pyhd8ed1ab_0
  - matplotlib-base=3.5.0=py39h2fa2bec_0
  - mccabe=0.6.1=py_1
  - mkl=2021.4.0=h8d4b97c_729
  - mkl-devel=2021.4.0=ha770c72_730
  - mkl-include=2021.4.0=h8d4b97c_729
  - more-itertools=8.11.0=pyhd8ed1ab_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - mypy=0.910=py39h3811e60_3
  - mypy_extensions=0.4.3=py39hf3d152e_4
  - nbformat=5.1.3=pyhd8ed1ab_0
  - ncurses=6.2=h58526e2_4
  - nettle=3.6=he412f7d_0
  - networkx=2.6.3=pyhd8ed1ab_1
  - nodeenv=1.6.0=pyhd8ed1ab_0
  - numpy=1.19.5=py39hdbf815f_2
  - olefile=0.46=pyh9f0ad1d_1
  - openh264=2.1.1=h780b84a_0
  - openjpeg=2.4.0=hb52868f_1
  - openssl=1.1.1l=h7f98852_0
  - packaging=21.3=pyhd8ed1ab_0
  - pandas=1.3.4=py39hde0f152_1
  - partd=1.2.0=pyhd8ed1ab_0
  - patchelf=0.13=h58526e2_0
  - pathspec=0.9.0=pyhd8ed1ab_0
  - pillow=8.4.0=py39ha612740_0
  - pip=21.3.1=pyhd8ed1ab_0
  - pkginfo=1.8.1=pyhd8ed1ab_0
  - platformdirs=2.3.0=pyhd8ed1ab_0
  - pluggy=1.0.0=py39hf3d152e_2
  - pooch=1.5.2=pyhd8ed1ab_0
  - pre-commit=2.15.0=py39hf3d152e_1
  - pre_commit=2.15.0=hd8ed1ab_1
  - protobuf=3.19.1=py39he80948d_1
  - psutil=5.8.0=py39h3811e60_2
  - py=1.11.0=pyh6c4a22f_0
  - py-lief=0.11.5=py39he80948d_1
  - pycodestyle=2.8.0=pyhd8ed1ab_0
  - pycosat=0.6.3=py39h3811e60_1009
  - pycparser=2.21=pyhd8ed1ab_0
  - pyflakes=2.4.0=pyhd8ed1ab_0
  - pyopenssl=21.0.0=pyhd8ed1ab_0
  - pyparsing=3.0.6=pyhd8ed1ab_0
  - pyrsistent=0.18.0=py39h3811e60_0
  - pysocks=1.7.1=py39hf3d152e_4
  - pytest=6.2.5=py39hf3d152e_1
  - pytest-grpc=0.8.0=py_0
  - python=3.9.7=hb7a2778_3_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-libarchive-c=3.1=py39hf3d152e_1
  - python_abi=3.9=2_cp39
  - pytorch=1.10.0=py3.9_cpu_0
  - pytorch-mutex=1.0=cpu
  - pytz=2021.3=pyhd8ed1ab_0
  - pywavelets=1.2.0=py39hce5d2b2_0
  - pyyaml=5.3.1=py39h3811e60_2
  - readline=8.1=h46c0cb4_0
  - regex=2021.11.10=py39h3811e60_0
  - requests=2.26.0=pyhd8ed1ab_0
  - ripgrep=13.0.0=habb4d0f_1
  - ruamel.yaml=0.17.17=py39h3811e60_1
  - ruamel.yaml.clib=0.2.6=py39h3811e60_0
  - ruamel_yaml=0.15.80=py39h3811e60_1006
  - scikit-image=0.18.3=py39hde0f152_0
  - scikit-learn=1.0.1=py39h4dfa638_2
  - scipy=1.7.2=py39hee8e79c_0
  - setuptools=59.2.0=py39hf3d152e_0
  - six=1.15.0=pyh9f0ad1d_0
  - snappy=1.1.8=he1b5a44_3
  - soupsieve=2.3=pyhd8ed1ab_0
  - sqlite=3.36.0=h9cd32fc_2
  - tbb=2021.4.0=h4bd325d_1
  - tensorboardx=2.4=pyhd8ed1ab_0
  - threadpoolctl=3.0.0=pyh8a188c0_0
  - tifffile=2021.11.2=pyhd8ed1ab_0
  - tk=8.6.11=h27826a3_1
  - toml=0.10.2=pyhd8ed1ab_0
  - tomli=1.2.2=pyhd8ed1ab_0
  - toolz=0.11.2=pyhd8ed1ab_0
  - torchvision=0.11.1=py39_cpu
  - tqdm=4.62.3=pyhd8ed1ab_0
  - traitlets=5.1.1=pyhd8ed1ab_0
  - typed-ast=1.5.0=py39h3811e60_0
  - typing-extensions=4.0.0=hd8ed1ab_0
  - typing_extensions=4.0.0=pyha770c72_0
  - tzdata=2021e=he74cb21_0
  - urllib3=1.26.7=pyhd8ed1ab_0
  - virtualenv=20.4.7=py39hf3d152e_1
  - wheel=0.37.0=pyhd8ed1ab_1
  - xarray=0.20.1=pyhd8ed1ab_0
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - zfp=0.5.5=h9c3ff4c_7
  - zipp=3.6.0=pyhd8ed1ab_0
  - zlib=1.2.11=h36c2ea0_1013
  - zstd=1.5.0=ha95c52a_0

for now I opted to skip this test in CI, which is of course bad. Need to look into this.