Cutout.prepare() never completes

[Ovewh]

Cutout preparation ERA5 never completes.

Description

cutout = atlite.Cutout(path="europe_2011_1.nc", module='era5',
                       x=slice(-13,45),
                      y=slice(32,83),
                      time='2011-01-01')    
cutout.prepare() # Never completes

Expected Behavior

Download data and return prepared cutout

Actual Behavior

The prepare never finishes even though the ERA5 data get downloaded.

Error Message

I interrupted the program after ~20 minutes

2021-06-23 14:00:19,649 INFO Requesting data for feature wind...
2021-06-23 14:00:19,651 INFO Requesting data for feature influx...
2021-06-23 14:00:19,666 INFO NumExpr defaulting to 8 threads.
2021-06-23 14:00:19,913 INFO CDS: Downloading variables
	 * 2m_temperature (2011)
	 * soil_temperature_level_4 (2011)

2021-06-23 14:00:21,616 INFO CDS: Downloading variables
	 * runoff (2011)

2021-06-23 14:00:22,981 INFO CDS: Downloading variables
	 * surface_net_solar_radiation (2011)
	 * surface_solar_radiation_downwards (2011)
	 * toa_incident_solar_radiation (2011)
	 * total_sky_direct_solar_radiation_at_surface (2011)

2021-06-23 14:00:26,740 INFO CDS: Downloading variables
	 * 100m_u_component_of_wind (2011)
	 * 100m_v_component_of_wind (2011)
	 * forecast_surface_roughness (2011)

                                                    

---------------------------------------------------------------------------
KeyboardInterrupt                         Traceback (most recent call last)
<ipython-input-2-eee9441f41e9> in <module>
----> 1 cutout.prepare()

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/atlite/data.py in wrapper(*args, **kwargs)
    100             kwargs["tmpdir"] = mkdtemp()
    101             try:
--> 102                 res = func(*args, **kwargs)
    103             finally:
    104                 rmtree(kwargs["tmpdir"])

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/atlite/data.py in cutout_prepare(cutout, features, tmpdir, overwrite)
    162         logger.info(f"Calculating and writing with module {module}:")
    163         missing_features = missing_vars.index.unique("feature")
--> 164         ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
    165         prepared |= set(missing_features)
    166 

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/atlite/data.py in get_features(cutout, module, features, tmpdir)
     44         datasets.append(feature_data)
     45 
---> 46     datasets = compute(*datasets)
     47 
     48     ds = xr.merge(datasets, compat="equals")

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/dask/base.py in compute(*args, **kwargs)
    563         postcomputes.append(x.__dask_postcompute__())
    564 
--> 565     results = schedule(dsk, keys, **kwargs)
    566     return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
    567 

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/dask/threaded.py in get(dsk, result, cache, num_workers, pool, **kwargs)
     74                 pools[thread][num_workers] = pool
     75 
---> 76     results = get_async(
     77         pool.apply_async,
     78         len(pool._pool),

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/dask/local.py in get_async(apply_async, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, **kwargs)
    474             # Main loop, wait on tasks to finish, insert new ones
    475             while state["waiting"] or state["ready"] or state["running"]:
--> 476                 key, res_info, failed = queue_get(queue)
    477                 if failed:
    478                     exc, tb = loads(res_info)

~/miniconda3/envs/atlite_dev/lib/python3.9/site-packages/dask/local.py in queue_get(q)
    131 
    132     def queue_get(q):
--> 133         return q.get()
    134 
    135 

~/miniconda3/envs/atlite_dev/lib/python3.9/queue.py in get(self, block, timeout)
    169             elif timeout is None:
    170                 while not self._qsize():
--> 171                     self.not_empty.wait()
    172             elif timeout < 0:
    173                 raise ValueError("'timeout' must be a non-negative number")

~/miniconda3/envs/atlite_dev/lib/python3.9/threading.py in wait(self, timeout)
    310         try:    # restore state no matter what (e.g., KeyboardInterrupt)
    311             if timeout is None:
--> 312                 waiter.acquire()
    313                 gotit = True
    314             else:

KeyboardInterrupt: 

Your Environment

• The atlite version used:
• How you installed atlite (conda, pip or github): github
• Operating System: ubuntu
• My environment:
  name: atlite_dev
  channels:
  • conda-forge
  • defaults
    dependencies:
  • _libgcc_mutex=0.1
  • _openmp_mutex=4.5
  • affine=2.3.0
  • anyio=3.2.0
  • appdirs=1.4.4
  • argon2-cffi=20.1.0
  • async_generator=1.10
  • attrs=21.2.0
  • babel=2.9.1
  • backcall=0.2.0
  • backports=1.0
  • backports.functools_lru_cache=1.6.4
  • black=21.5b2
  • bleach=3.3.0
  • bokeh=2.3.2
  • boost-cpp=1.74.0
  • bottleneck=1.3.2
  • brotlipy=0.7.0
  • bzip2=1.0.8
  • c-ares=1.17.1
  • ca-certificates=2021.5.30
  • cairo=1.16.0
  • cdsapi=0.5.1
  • certifi=2021.5.30
  • cffi=1.14.5
  • cfgv=3.3.0
  • cfitsio=3.470
  • cftime=1.5.0
  • chardet=4.0.0
  • click=7.1.2
  • click-plugins=1.1.1
  • cligj=0.7.2
  • cloudpickle=1.6.0
  • cryptography=3.4.7
  • curl=7.77.0
  • cycler=0.10.0
  • cytoolz=0.11.0
  • dask=2021.3.1
  • dask-core=2021.3.1
  • dataclasses=0.8
  • decorator=5.0.9
  • defusedxml=0.7.1
  • distlib=0.3.2
  • distributed=2021.4.1
  • editdistance-s=1.0.0
  • entrypoints=0.3
  • esgf-pyclient=0.3.0
  • expat=2.4.1
  • filelock=3.0.12
  • fiona=1.8.20
  • fontconfig=2.13.1
  • freetype=2.10.4
  • freexl=1.0.6
  • fsspec=2021.6.0
  • gdal=3.2.2
  • geopandas=0.9.0
  • geopandas-base=0.9.0
  • geos=3.9.1
  • geotiff=1.6.0
  • gettext=0.19.8.1
  • giflib=5.2.1
  • hdf4=4.2.15
  • hdf5=1.10.6
  • heapdict=1.0.1
  • icu=68.1
  • identify=2.2.10
  • idna=2.10
  • importlib-metadata=4.5.0
  • iniconfig=1.1.1
  • ipykernel=5.5.5
  • ipython=7.24.1
  • ipython_genutils=0.2.0
  • jbig=2.1
  • jedi=0.18.0
  • jinja2=3.0.1
  • joblib=1.0.1
  • jpeg=9d
  • json-c=0.15
  • json5=0.9.5
  • jsonschema=3.2.0
  • jupyter_client=6.1.12
  • jupyter_core=4.7.1
  • jupyter_server=1.8.0
  • jupyterlab=3.0.16
  • jupyterlab_pygments=0.1.2
  • jupyterlab_server=2.6.0
  • kealib=1.4.14
  • kiwisolver=1.3.1
  • krb5=1.19.1
  • lcms2=2.12
  • ld_impl_linux-64=2.35.1
  • lerc=2.2.1
  • libblas=3.9.0
  • libcblas=3.9.0
  • libcurl=7.77.0
  • libdap4=3.20.6
  • libdeflate=1.7
  • libedit=3.1.20191231
  • libev=4.33
  • libffi=3.3
  • libgcc-ng=9.3.0
  • libgdal=3.2.2
  • libgfortran-ng=9.3.0
  • libgfortran5=9.3.0
  • libglib=2.68.3
  • libgomp=9.3.0
  • libiconv=1.16
  • libkml=1.3.0
  • liblapack=3.9.0
  • libnetcdf=4.8.0
  • libnghttp2=1.43.0
  • libopenblas=0.3.15
  • libpng=1.6.37
  • libpq=13.3
  • librttopo=1.1.0
  • libsodium=1.0.18
  • libspatialindex=1.9.3
  • libspatialite=5.0.1
  • libssh2=1.9.0
  • libstdcxx-ng=9.3.0
  • libtiff=4.3.0
  • libuuid=2.32.1
  • libwebp-base=1.2.0
  • libxcb=1.13
  • libxml2=2.9.12
  • libzip=1.8.0
  • locket=0.2.0
  • lz4-c=1.9.3
  • mapclassify=2.4.2
  • markupsafe=2.0.1
  • matplotlib-base=3.4.2
  • matplotlib-inline=0.1.2
  • mistune=0.8.4
  • more-itertools=8.8.0
  • msgpack-python=1.0.2
  • munch=2.5.0
  • mypy_extensions=0.4.3
  • nbclassic=0.3.1
  • nbclient=0.5.3
  • nbconvert=6.0.7
  • nbformat=5.1.3
  • ncurses=6.2
  • nest-asyncio=1.5.1
  • netcdf4=1.5.7
  • networkx=2.5
  • nodeenv=1.6.0
  • notebook=6.4.0
  • numexpr=2.7.3
  • numpy=1.21.0
  • olefile=0.46
  • openjpeg=2.4.0
  • openssl=1.1.1k
  • packaging=20.9
  • pandas=1.2.5
  • pandoc=2.14.0.2
  • pandocfilters=1.4.2
  • parso=0.8.2
  • partd=1.2.0
  • pathspec=0.8.1
  • pcre=8.45
  • pexpect=4.8.0
  • pickleshare=0.7.5
  • pillow=8.2.0
  • pip=21.1.2
  • pixman=0.40.0
  • pluggy=0.13.1
  • poppler=21.03.0
  • poppler-data=0.4.10
  • postgresql=13.3
  • pre-commit=2.13.0
  • progressbar2=3.53.1
  • proj=8.0.1
  • prometheus_client=0.11.0
  • prompt-toolkit=3.0.19
  • psutil=5.8.0
  • pthread-stubs=0.4
  • ptyprocess=0.7.0
  • py=1.10.0
  • pycparser=2.20
  • pygments=2.9.0
  • pyopenssl=20.0.1
  • pyparsing=2.4.7
  • pyproj=3.1.0
  • pyrsistent=0.17.3
  • pysocks=1.7.1
  • pytest=6.2.4
  • python=3.9.5
  • python-dateutil=2.8.1
  • python-utils=2.5.6
  • python_abi=3.9
  • pytz=2021.1
  • pyyaml=5.4.1
  • pyzmq=22.1.0
  • rasterio=1.2.5
  • readline=8.1
  • regex=2021.4.4
  • requests=2.25.1
  • requests_cache=0.4.13
  • rtree=0.9.7
  • scikit-learn=0.24.2
  • scipy=1.6.3
  • send2trash=1.7.1
  • setuptools=49.6.0
  • shapely=1.7.1
  • six=1.16.0
  • sniffio=1.2.0
  • snuggs=1.4.7
  • sortedcontainers=2.4.0
  • sqlite=3.36.0
  • tblib=1.7.0
  • terminado=0.10.1
  • testpath=0.5.0
  • threadpoolctl=2.1.0
  • tiledb=2.3.1
  • tk=8.6.10
  • toml=0.10.2
  • toolz=0.11.1
  • tornado=6.1
  • tqdm=4.61.1
  • traitlets=5.0.5
  • typed-ast=1.4.3
  • typing_extensions=3.10.0.0
  • tzcode=2021a
  • tzdata=2021a
  • urllib3=1.26.5
  • virtualenv=20.4.7
  • wcwidth=0.2.5
  • webencodings=0.5.1
  • webob=1.8.7
  • websocket-client=0.57.0
  • wheel=0.36.2
  • xarray=0.18.2
  • xerces-c=3.2.3
  • xorg-kbproto=1.0.7
  • xorg-libice=1.0.10
  • xorg-libsm=1.2.3
  • xorg-libx11=1.7.2
  • xorg-libxau=1.0.9
  • xorg-libxdmcp=1.1.3
  • xorg-libxext=1.3.4
  • xorg-libxrender=0.9.10
  • xorg-renderproto=0.11.1
  • xorg-xextproto=7.3.0
  • xorg-xproto=7.0.31
  • xz=5.2.5
  • yaml=0.2.5
  • zeromq=4.3.4
  • zict=2.0.0
  • zipp=3.4.1
  • zlib=1.2.11
  • zstd=1.5.0
  • pip:
    • atlite==0.2.5.dev23+g9990b1f

[FabianHofmann]

Hey @Ovewh, thanks for reporting. I cannot reproduce the error (testing on python 3.8).

Two possibilities:

1. Your request queues on the cds server (please check!)
2. Worse case: Bad communication between xarray and dask in python 3.9. We saw something similar in python 3.7. It happened rarely, but it happened. In this case, I cannot come up with a solution, since this goes very deep down into dask (it is also not a bug in atlite but in the communication between xarray and dask). Restarting and hoping the error does not occur again...

Note: when testing I saw another possible problem, which you might run in see #176

[Ovewh]

Thanks!
@FabianHofmann I tested only preparing a single feature at the time and that works fine. However, when I try to prepare the height my request returns no data. So I imagine that #176 would solve my problem.

[FabianHofmann]

#176 is now merged, could you try again on the up-to-date master?

[FabianHofmann]

please reopen, if there is still a related problem