What happened:
Dask scheduler is timing out
What you expected to happen:
Return result
Minimal Complete Verifiable Example:
from xxxxx.distributed.ecscluster import EcsCluster
from distributed import Client
e = EcsCluster(gpu=True)
client = Client(e.scheduler_address)
def fn():
return 9999
client.submit(fn).result()
Anything else we need to know?:
The scheduler is running as a fargate task in an ECS cluster. Workers are ECS tasks using EC2
It keeps running garbage collection when there are no jobs running
Error log
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
--
distributed.utils_perf - INFO - full garbage collection released 95.23 MiB from 149 reference cycles (threshold: 9.54 MiB)
distributed.scheduler - INFO - Remove client Client-b1e758c6-f480-11eb-a64e-06e9f6cd33c1
distributed.scheduler - INFO - Remove client Client-b1e758c6-f480-11eb-a64e-06e9f6cd33c1
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.scheduler - INFO - Close client connection: Client-b1e758c6-f480-11eb-a64e-06e9f6cd33c1
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.scheduler - INFO - Receive client connection: Client-d7f19254-f481-11eb-93cd-acde48001122
distributed.core - INFO - Starting established connection
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.scheduler - INFO - Remove client Client-d7f19254-f481-11eb-93cd-acde48001122
distributed.scheduler - INFO - Remove client Client-d7f19254-f481-11eb-93cd-acde48001122
distributed.scheduler - INFO - Close client connection: Client-d7f19254-f481-11eb-93cd-acde48001122
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.utils_perf - WARNING - full garbage collections took 26% CPU time recently (threshold: 10%)
distributed.scheduler - INFO - End scheduler at 'tcp://10.72.100.11:28786'
Traceback (most recent call last):
File "/opt/conda/bin/dask-scheduler", line 11, in <module>
sys.exit(go())
File "/opt/conda/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 218, in go
main()
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/conda/lib/python3.7/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/distributed/cli/dask_scheduler.py", line 209, in main
loop.run_sync(run)
File "/opt/conda/lib/python3.7/site-packages/tornado/ioloop.py", line 529, in run_sync
raise TimeoutError("Operation timed out after %s seconds" % timeout)
tornado.util.TimeoutError: Operation timed out after None seconds
distributed.utils_perf - WARNING - full garbage collections took 27% CPU time recently (threshold: 10%)
Environment:
- Dask version: 2021.4.0
- Python version: 3.7.19
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): conda install
conda list
# packages in environment at /opt/conda:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
abseil-cpp 20210324.1 h9c3ff4c_0 conda-forge
absl-py 0.13.0 pypi_0 pypi
aiobotocore 1.3.3 pypi_0 pypi
aiohttp 3.7.4.post0 py37h5e8e339_0 conda-forge
aioitertools 0.7.1 pypi_0 pypi
anndata 0.7.6 pypi_0 pypi
annoy 1.17.0 pypi_0 pypi
ansiwrap 0.8.4 pypi_0 pypi
anyio 3.3.0 py37h89c1867_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 20.1.0 py37h5e8e339_2 conda-forge
arrow-cpp 1.0.1 py37haa335b2_40_cuda conda-forge
arrow-cpp-proc 3.0.0 cuda conda-forge
astor 0.8.1 pypi_0 pypi
astroid 2.3.3 pypi_0 pypi
async-timeout 3.0.1 py_1000 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
aws-c-cal 0.5.11 h95a6274_0 conda-forge
aws-c-common 0.6.2 h7f98852_0 conda-forge
aws-c-event-stream 0.2.7 h3541f99_13 conda-forge
aws-c-io 0.10.5 hfb6a706_0 conda-forge
aws-checksums 0.1.11 ha31a3da_7 conda-forge
aws-requests-auth 0.4.3 pypi_0 pypi
aws-sdk-cpp 1.8.186 hb4091e7_3 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backoff 1.11.1 pypi_0 pypi
backports 1.0 py_2 conda-forge
backports-entry-points-selectable 1.1.0 pypi_0 pypi
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bbknn 1.5.1 pypi_0 pypi
black 21.7b0 pypi_0 pypi
bleach 3.3.1 pyhd8ed1ab_0 conda-forge
blosc 1.9.1 pypi_0 pypi
bokeh 2.2.3 py37h89c1867_0 conda-forge
boost 1.72.0 py37h48f8a5e_1 conda-forge
boost-cpp 1.72.0 h9d3c048_4 conda-forge
boto3 1.18.10 pypi_0 pypi
botocore 1.21.10 pypi_0 pypi
brotli 1.0.9 h7f98852_5 conda-forge
brotli-bin 1.0.9 h7f98852_5 conda-forge
brotlipy 0.7.0 py37h27cfd23_1003
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h7f98852_1 conda-forge
ca-certificates 2021.7.5 h06a4308_1
cachetools 4.2.2 pyhd8ed1ab_0 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
certifi 2021.5.30 py37h06a4308_0
cffi 1.14.6 py37h400218f_0
cfgv 3.3.0 pypi_0 pypi
cfitsio 3.470 hb418390_7 conda-forge
chardet 4.0.0 py37h06a4308_1003
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_0 conda-forge
cloudpickle 1.6.0 py_0 conda-forge
cmake 3.17.2 pypi_0 pypi
colorama 0.4.4 pypi_0 pypi
colorcet 2.0.6 pyhd8ed1ab_0 conda-forge
commonmark 0.9.1 pypi_0 pypi
conda 4.10.3 py37h06a4308_0
conda-package-handling 1.7.3 py37h27cfd23_1
coverage 5.5 pypi_0 pypi
cryptography 3.4.7 py37hd23ed53_0
cudatoolkit 10.2.89 h8f6ccaa_8 nvidia
cudf 0.19.2 cuda_10.2_py37_gab3b3f653a_0 rapidsai
cudf_kafka 0.19.2 py37_gab3b3f653a_0 rapidsai
cudnn 8.2.1.32 h2c0ae14_0 conda-forge
cugraph 0.19.0 py37_gd72b90b0_0 rapidsai
cuml 0.19.0 cuda10.2_py37_g4cb78ff1a_0 rapidsai
cupy 8.6.0 py37hfed0110_0 conda-forge
curl 7.78.0 hea6ffbf_0 conda-forge
cusignal 0.19.0 py38_g347b95a_0 rapidsai
cuspatial 0.19.0 py37_gdf1d93c_0 rapidsai
custreamz 0.19.2 py37_gab3b3f653a_0 rapidsai
cutensor 1.2.2.5 h1a5f58c_5 conda-forge
cuxfilter 0.19.1 py37_g82dc80c_0 rapidsai
cycler 0.10.0 py_2 conda-forge
cyrus-sasl 2.1.27 h230043b_2 conda-forge
cython 0.29.24 pypi_0 pypi
cytoolz 0.11.0 py37h5e8e339_3 conda-forge
dask 2021.4.0 pyhd8ed1ab_0 conda-forge
dask-core 2021.4.0 pyhd8ed1ab_0 conda-forge
dask-cuda 0.19.0 py37_0 rapidsai
dask-cudf 0.19.2 py37_gab3b3f653a_0 rapidsai
dataclasses 0.6 pypi_0 pypi
datashader 0.11.1 pyh9f0ad1d_0 conda-forge
datashape 0.5.4 py_1 conda-forge
dateutils 0.6.12 pypi_0 pypi
decorator 4.4.2 py_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
distlib 0.3.2 pypi_0 pypi
distributed 2021.4.0 py37h89c1867_0 conda-forge
dlpack 0.3 he1b5a44_1 conda-forge
dnspython 2.1.0 pypi_0 pypi
egnyte 0.5.3 pypi_0 pypi
elasticsearch 7.13.4 pypi_0 pypi
entrypoints 0.3 pyhd8ed1ab_1003 conda-forge
et-xmlfile 1.1.0 pypi_0 pypi
expat 2.4.1 h9c3ff4c_0 conda-forge
faiss-proc 1.0.0 cuda rapidsai
fastavro 1.4.4 py37h5e8e339_0 conda-forge
fastrlock 0.6 py37hcd2ae1e_1 conda-forge
fcsparser 0.2.1 pypi_0 pypi
filelock 3.0.12 pypi_0 pypi
fiona 1.8.20 py37ha0cc35a_0 conda-forge
flask 2.0.1 pypi_0 pypi
flask-cors 3.0.10 pypi_0 pypi
flask-json 0.3.4 pypi_0 pypi
fontconfig 2.13.1 hba837de_1005 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
fsspec 0.8.5 pypi_0 pypi
future 0.18.2 pypi_0 pypi
gast 0.2.2 pypi_0 pypi
gdal 3.2.2 py37hb0e9ad2_5 conda-forge
geopandas 0.8.1 py_0 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
geotiff 1.6.0 hcf90da6_5 conda-forge
gettext 0.19.8.1 h0b5b191_1005 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
glog 0.5.0 h48cff8f_0 conda-forge
google-api-core 1.31.1 pypi_0 pypi
google-auth 1.34.0 pypi_0 pypi
google-cloud-core 1.7.2 pypi_0 pypi
google-cloud-storage 1.25.0 pypi_0 pypi
google-pasta 0.2.0 pypi_0 pypi
google-resumable-media 0.5.1 pypi_0 pypi
googleapis-common-protos 1.53.0 pypi_0 pypi
grpc-cpp 1.38.0 h2519f57_0 conda-forge
grpcio 1.39.0 pypi_0 pypi
h5py 2.10.0 pypi_0 pypi
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1114 conda-forge
heapdict 1.0.1 py_0 conda-forge
hnswlib 0.5.2 pypi_0 pypi
hotspot 0.9.1 pypi_0 pypi
hyperopt 0.1.2 pypi_0 pypi
icu 68.1 h58526e2_0 conda-forge
identify 2.2.11 pypi_0 pypi
idna 2.10 pyhd3eb1b0_0
imageio 2.9.0 pypi_0 pypi
importlib-metadata 1.7.0 pypi_0 pypi
importlib-resources 5.2.1 pypi_0 pypi
iniconfig 1.1.1 pypi_0 pypi
ipykernel 5.5.5 py37h085eea5_0 conda-forge
ipython 7.25.0 py37h085eea5_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.3 pyhd3deb0d_0 conda-forge
isort 4.3.21 pypi_0 pypi
itsdangerous 2.0.1 pypi_0 pypi
jedi 0.18.0 py37h89c1867_2 conda-forge
jinja2 3.0.1 pyhd8ed1ab_0 conda-forge
jmespath 0.10.0 pypi_0 pypi
joblib 1.0.1 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
jsonlines 1.2.0 pypi_0 pypi
jsonpickle 2.0.0 pypi_0 pypi
jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge
jupyter 1.0.0 pypi_0 pypi
jupyter-console 6.4.0 pypi_0 pypi
jupyter-contrib-core 0.3.3 pypi_0 pypi
jupyter-contrib-nbextensions 0.5.1 pypi_0 pypi
jupyter-highlight-selected-word 0.2.0 pypi_0 pypi
jupyter-latex-envs 1.4.6 pypi_0 pypi
jupyter-nbextensions-configurator 0.4.1 pypi_0 pypi
jupyter-server-proxy 3.1.0 pyhd8ed1ab_0 conda-forge
jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge
jupyter_core 4.7.1 py37h89c1867_0 conda-forge
jupyter_server 1.10.1 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
kealib 1.4.14 hcc255d8_2 conda-forge
keras-applications 1.0.8 pypi_0 pypi
keras-preprocessing 1.1.2 pypi_0 pypi
kiwisolver 1.3.1 py37h2527ec5_1 conda-forge
kneed 0.7.0 pypi_0 pypi
krb5 1.19.2 hcc1bbae_0 conda-forge
lazy-object-proxy 1.4.3 pypi_0 pypi
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.35.1 h7274673_9
leidenalg 0.7.0 pypi_0 pypi
libblas 3.9.0 10_openblas conda-forge
libbrotlicommon 1.0.9 h7f98852_5 conda-forge
libbrotlidec 1.0.9 h7f98852_5 conda-forge
libbrotlienc 1.0.9 h7f98852_5 conda-forge
libcblas 3.9.0 10_openblas conda-forge
libcudf 0.19.2 cuda10.2_gab3b3f653a_0 rapidsai
libcudf_kafka 0.19.2 gab3b3f653a_0 rapidsai
libcugraph 0.19.0 cuda10.2_gd72b90b0_0 rapidsai
libcuml 0.19.0 cuda10.2_g4cb78ff1a_0 rapidsai
libcumlprims 0.19.0 cuda10.2_ga2abf9f_0 nvidia
libcurl 7.78.0 h2574ce0_0 conda-forge
libcuspatial 0.19.0 cuda10.2_gdf1d93c_0 rapidsai
libdap4 3.20.6 hd7c4107_2 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libfaiss 1.7.0 cuda102hd37495c_8_cuda conda-forge
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgcrypt 1.9.3 h7f98852_1 conda-forge
libgdal 3.2.2 h679344c_5 conda-forge
libgfortran-ng 11.1.0 h69a702a_4 conda-forge
libgfortran5 11.1.0 h6c583b3_4 conda-forge
libglib 2.68.3 h3e27bee_0 conda-forge
libgomp 9.3.0 h5101ec6_17
libgpg-error 1.42 h9c3ff4c_0 conda-forge
libgsasl 1.8.0 2 conda-forge
libhwloc 2.3.0 h5e5b7d1_1 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 hd79254b_1012 conda-forge
liblapack 3.9.0 10_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libnetcdf 4.8.0 nompi_hcd642e3_103 conda-forge
libnghttp2 1.43.0 h812cca2_0 conda-forge
libntlm 1.4 h7f98852_1002 conda-forge
libopenblas 0.3.17 pthreads_h8fe5266_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.3 hd57d9b9_0 conda-forge
libprotobuf 3.16.0 h780b84a_0 conda-forge
librdkafka 1.5.3 hc49e61c_1 conda-forge
librmm 0.19.0 cuda10.2_g7065af3_0 rapidsai
librttopo 1.1.0 h1185371_6 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.0.1 h20cb978_4 conda-forge
libssh2 1.9.0 ha56f1ee_6 conda-forge
libstdcxx-ng 9.3.0 hd4cf53a_17
libthrift 0.14.1 he6d91bd_2 conda-forge
libtiff 4.2.0 hbd63e13_2 conda-forge
libutf8proc 2.6.1 h7f98852_0 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libuv 1.42.0 h7f98852_0 conda-forge
libwebp 1.2.0 h3452ae3_0 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h7f98852_1003 conda-forge
libxgboost 1.4.0dev.rapidsai0.19 cuda10.2_0 rapidsai
libxml2 2.9.12 h72842e0_0 conda-forge
libzip 1.8.0 h4de3113_0 conda-forge
llvmlite 0.35.0 py37h612dafd_4
locket 0.2.0 py_2 conda-forge
louvain 0.7.0 pypi_0 pypi
lxml 4.6.3 pypi_0 pypi
lz4 3.1.0 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
markdown 3.3.4 pyhd8ed1ab_0 conda-forge
markupsafe 2.0.1 py37h5e8e339_0 conda-forge
matplotlib 3.2.1 pypi_0 pypi
matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge
mccabe 0.6.1 pypi_0 pypi
metaflow 2.3.2 pypi_0 pypi
mistune 0.8.4 py37h5e8e339_1004 conda-forge
msgpack-python 1.0.2 py37h2527ec5_1 conda-forge
multidict 5.1.0 py37h5e8e339_1 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
munch 2.5.0 py_0 conda-forge
mypy-extensions 0.4.3 pypi_0 pypi
natsort 7.1.1 pypi_0 pypi
nbclient 0.5.3 pyhd8ed1ab_0 conda-forge
nbconvert 5.6.1 pypi_0 pypi
nbformat 5.0.8 pypi_0 pypi
nbval 0.9.6 pypi_0 pypi
nccl 2.10.3.1 h1a5f58c_0 conda-forge
ncurses 6.2 he6710b0_1
nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge
networkx 2.6.2 pyhd8ed1ab_0 conda-forge
nodeenv 1.6.0 pypi_0 pypi
nodejs 14.17.4 h92b4a50_0 conda-forge
notebook 6.4.0 pyha770c72_0 conda-forge
numba 0.52.0 py37ha9443f7_0
numexpr 2.7.3 pypi_0 pypi
numpy 1.18.5 pypi_0 pypi
nvtx 0.2.3 py37h5e8e339_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
opencv-python 4.5.3.56 pypi_0 pypi
openjpeg 2.4.0 hb52868f_1 conda-forge
openpyxl 3.0.7 pypi_0 pypi
openssl 1.1.1k h27cfd23_0
opentsne 0.6.0 pypi_0 pypi
opt-einsum 3.3.0 pypi_0 pypi
orc 1.6.7 h89a63ab_2 conda-forge
packaging 21.0 pyhd8ed1ab_0 conda-forge
pandas 1.2.4 py37h219a48f_0 conda-forge
pandoc 2.14.1 h7f98852_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
panel 0.10.3 pyhd8ed1ab_0 conda-forge
papermill 2.2.2 pypi_0 pypi
param 1.11.1 pyh6c4a22f_0 conda-forge
parc 0.31 pypi_0 pypi
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.2 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pathspec 0.9.0 pypi_0 pypi
patsy 0.5.1 pypi_0 pypi
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickle5 0.0.11 py37h5e8e339_0 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.2.0 py37h4600e1f_1 conda-forge
pip 21.1.3 py37h06a4308_0
pixman 0.40.0 h36c2ea0_0 conda-forge
platformdirs 2.2.0 pypi_0 pypi
pluggy 0.13.1 pypi_0 pypi
poppler 21.03.0 h93df280_0 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 13.3 h2510834_0 conda-forge
pre-commit 2.13.0 pypi_0 pypi
proj 8.0.0 h277dcde_0 conda-forge
prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.19 pyha770c72_0 conda-forge
protobuf 3.16.0 py37hcd2ae1e_0 conda-forge
psutil 5.8.0 py37h5e8e339_1 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
py 1.10.0 pypi_0 pypi
py-xgboost 1.4.0dev.rapidsai0.19 cuda10.2py37_0 rapidsai
pyarrow 1.0.1 py37hb63ea2f_40_cuda conda-forge
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pybind11 2.7.0 pypi_0 pypi
pybiomart 0.2.0 pypi_0 pypi
pycosat 0.6.3 py37h27cfd23_0
pycparser 2.20 py_2
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pydeck 0.5.0 pyh9f0ad1d_0 conda-forge
pyee 7.0.4 pyh9f0ad1d_0 conda-forge
pyfaidx 0.6.1 pypi_0 pypi
pygments 2.9.0 pyhd8ed1ab_0 conda-forge
pylint 2.4.4 pypi_0 pypi
pymongo 3.12.0 pypi_0 pypi
pynndescent 0.5.4 pypi_0 pypi
pynvml 11.0.0 pyhd8ed1ab_0 conda-forge
pyopenssl 20.0.1 pyhd3eb1b0_1
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyppeteer 0.2.2 py_1 conda-forge
pyproj 3.0.1 py37h2bb2a07_1 conda-forge
pyrsistent 0.17.3 py37h5e8e339_2 conda-forge
pysam 0.16.0.1 pypi_0 pypi
pysocks 1.7.1 py37_1
pytest 6.2.4 pypi_0 pypi
pytest-cov 2.12.1 pypi_0 pypi
python 3.7.10 hffdb5ce_100_cpython conda-forge
python-box 5.3.0 pypi_0 pypi
python-confluent-kafka 1.5.0 py37h8f50634_0 conda-forge
python-dateutil 2.7.5 pypi_0 pypi
python-igraph 0.9.6 pypi_0 pypi
python_abi 3.7 2_cp37m conda-forge
pytz 2021.1 pyhd8ed1ab_0 conda-forge
pyviz_comms 2.1.0 pyhd8ed1ab_0 conda-forge
pywavelets 1.1.1 pypi_0 pypi
pyyaml 5.4.1 py37h27cfd23_1
pyzmq 22.1.0 py37h336d617_0 conda-forge
qtconsole 5.1.1 pypi_0 pypi
qtpy 1.9.0 pypi_0 pypi
quilt3 3.4.0 pypi_0 pypi
rapids 0.19.0 cuda10.2_py37_ga03647d_299 rapidsai
rapids-xgboost 0.19.0 cuda10.2_py37_ga03647d_299 rapidsai
re2 2021.04.01 h9c3ff4c_0 conda-forge
readline 8.1 h27cfd23_0
regex 2021.7.6 pypi_0 pypi
requests 2.25.1 pyhd3eb1b0_0
requests-cache 0.7.2 pypi_0 pypi
requests-futures 1.0.0 pypi_0 pypi
requests-unixsocket 0.2.0 py_0 conda-forge
retrying 1.3.3 pypi_0 pypi
rich 10.6.0 pypi_0 pypi
rmm 0.19.0 cuda_10.2_py37_g7065af3_0 rapidsai
rsa 4.7.2 pypi_0 pypi
rtree 0.9.7 py37h0b55af0_2 conda-forge
ruamel-yaml 0.17.10 pypi_0 pypi
ruamel-yaml-clib 0.2.6 pypi_0 pypi
ruamel_yaml 0.15.100 py37h27cfd23_0
s2n 1.0.10 h9b69904_0 conda-forge
s3fs 0.5.2 pypi_0 pypi
s3transfer 0.5.0 pypi_0 pypi
scanpy 1.8.1 pypi_0 pypi
scikit-image 0.18.2 pypi_0 pypi
scikit-learn 0.24.2 py37h18a542f_0 conda-forge
scikit-misc 0.1.4 pypi_0 pypi
scipy 1.7.0 py37h29e03ee_1 conda-forge
scrublet 0.2.1 pypi_0 pypi
scvi-tools 0.8.1 pypi_0 pypi
seaborn 0.11.0 pypi_0 pypi
send2trash 1.7.1 pyhd8ed1ab_0 conda-forge
setuptools 49.6.0 py37h89c1867_3 conda-forge
shapely 1.7.1 py37h2d1e849_5 conda-forge
simpervisor 0.4 pyhd8ed1ab_0 conda-forge
sinfo 0.3.4 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_0
sklearn 0.0 pypi_0 pypi
slackclient 1.0.0 pypi_0 pypi
snappy 1.1.8 he1b5a44_3 conda-forge
sniffio 1.2.0 py37h89c1867_1 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sparse 0.12.0 pypi_0 pypi
spdlog 1.7.0 hc9558a2_2 conda-forge
sqlite 3.36.0 hc218d9a_0
statsmodels 0.12.2 pypi_0 pypi
stdlib-list 0.8.0 pypi_0 pypi
streamz 0.6.2 pyh44b312d_0 conda-forge
tables 3.6.1 pypi_0 pypi
tbb 2021.3.0 pypi_0 pypi
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tenacity 8.0.1 pypi_0 pypi
tensorboard 1.15.0 pypi_0 pypi
tensorflow 1.15.5 pypi_0 pypi
tensorflow-estimator 1.15.1 pypi_0 pypi
termcolor 1.1.0 pypi_0 pypi
terminado 0.10.1 py37h89c1867_0 conda-forge
testpath 0.5.0 pyhd8ed1ab_0 conda-forge
texttable 1.6.4 pypi_0 pypi
textwrap3 0.9.2 pypi_0 pypi
threadpoolctl 2.2.0 pyh8a188c0_0 conda-forge
tifffile 2021.7.30 pypi_0 pypi
tiledb 2.2.9 h91fcb0e_0 conda-forge
tk 8.6.10 hbc83047_0
toml 0.10.0 pypi_0 pypi
tomli 1.2.0 pypi_0 pypi
toolz 0.11.1 py_0 conda-forge
torch 1.9.0 pypi_0 pypi
tornado 6.1 py37h5e8e339_1 conda-forge
tqdm 4.61.2 pyhd3eb1b0_1
traitlets 5.0.5 py_0 conda-forge
treelite 1.1.0 py37hc731546_0 conda-forge
treelite-runtime 1.1.0 pypi_0 pypi
typed-ast 1.4.3 pypi_0 pypi
typing-extensions 3.10.0.0 hd8ed1ab_0 conda-forge
typing_extensions 3.10.0.0 pyha770c72_0 conda-forge
tzcode 2021a h7f98852_2 conda-forge
tzdata 2021a he74cb21_1 conda-forge
ucx 1.9.0+gcd9efd3 cuda10.2_0 rapidsai
ucx-proc 1.0.0 gpu rapidsai
ucx-py 0.19.0 py37_gcd9efd3_0 rapidsai
umap-learn 0.4.2 pypi_0 pypi
url-normalize 1.4.3 pypi_0 pypi
urllib3 1.26.6 pyhd3eb1b0_1
virtualenv 20.6.0 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 0.57.0 py37h89c1867_4 conda-forge
websockets 8.1 py37h5e8e339_3 conda-forge
werkzeug 2.0.1 pypi_0 pypi
wheel 0.36.2 pyhd3eb1b0_0
widgetsnbextension 3.5.1 py37h89c1867_4 conda-forge
wrapt 1.11.2 pypi_0 pypi
xarray 0.19.0 pyhd8ed1ab_1 conda-forge
xerces-c 3.2.3 h9d8b166_2 conda-forge
xgboost 1.4.0dev.rapidsai0.19 cuda10.2py37_0 rapidsai
xlrd 1.2.0 pypi_0 pypi
xlwt 1.3.0 pypi_0 pypi
xmltodict 0.12.0 pypi_0 pypi
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
yarl 1.6.3 py37h5e8e339_2 conda-forge
zeromq 4.3.4 h9c3ff4c_0 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.5.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 ha95c52a_0 conda-forge
What happened:
Dask scheduler is timing out
What you expected to happen:
Return result
Minimal Complete Verifiable Example:
Anything else we need to know?:
The scheduler is running as a fargate task in an ECS cluster. Workers are ECS tasks using EC2
It keeps running garbage collection when there are no jobs running
Error log
Environment:
conda list