Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Chunking works only with dask installed #50

Open
corrado9999 opened this issue Apr 27, 2021 · 5 comments
Open

Chunking works only with dask installed #50

corrado9999 opened this issue Apr 27, 2021 · 5 comments
Labels
wontfix This will not be worked on

Comments

@corrado9999
Copy link
Collaborator

I am not sure if this is a bug and, if so, of what.

>>> path = "tests/data/S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE/manifest.safe"
>>> group = 'IW1/R168-N471-E0118'
>>> xs.sentinel1.open_dataset(path, group=group)
<xarray.Dataset>
Dimensions:           (azimuth_time: 1501, slant_range_time: 21632)
Coordinates:
    line              (azimuth_time) float64 0.5 1.5 2.5 ... 1.5e+03 1.5e+03
    pixel             (slant_range_time) float64 0.5 1.5 ... 2.163e+04 2.163e+04
  * azimuth_time      (azimuth_time) datetime64[ns] 2021-04-01T05:26:24.20999...
  * slant_range_time  (slant_range_time) float64 0.005343 0.005343 ... 0.005679
Data variables:
    VH                (azimuth_time, slant_range_time) complex128 ...
    VV                (azimuth_time, slant_range_time) complex128 ...
Attributes: (12/14)
    constellation:              sentinel-1
    platform:                   sentinel-1b
    instrument:                 ['c-sar']
    sat:orbit_state:            descending
    sat:absolute_orbit:         26269
    sat:relative_orbit:         168
    ...                         ...
    sar:instrument_mode:        IW
    sar:polarizations:          ['VV', 'VH']
    sar:product_type:           SLC
    xs:instrument_mode_swaths:  ['IW1', 'IW2', 'IW3']
    Conventions:                CF-1.7
    history:                    created by xarray_sentinel-999

>>> xs.sentinel1.open_dataset(path, group=group, chunks=1000)
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-5-79175c680e53> in <module>
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-90f5963f6ea0> in <module>
----> 1 xs.sentinel1.open_dataset(path, group=group, chunks=1000)

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_dataset(product_urlpath, drop_variables, group, chunks, fs)
    384         else:
    385             annotation_path = list(groups[group]["annotation_path"].values())[0]
--> 386             ds = open_burst_dataset(
    387                 manifest_path,
    388                 measurement_paths=groups[group]["measurement_path"],

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_burst_dataset(manifest_path, burst_position, measurement_paths, annotation_path, chunks)
    297     data_vars = {}
    298     for pol, data_path in measurement_paths.items():
--> 299         arr = rioxarray.open_rasterio(data_path, chunks=chunks)
    300
    301         arr = arr.squeeze("band").drop_vars(["band", "spatial_ref"])

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, **open_kwargs)
    863
    864     if chunks is not None:
--> 865         result = _prepare_dask(result, riods, filename, chunks)
    866
    867     # Make the file closeable

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in _prepare_dask(result, riods, filename, chunks)
    573     """
    574     # pylint: disable=import-outside-toplevel
--> 575     from dask.base import tokenize
    576
    577     # augment the token with the file modification time

ModuleNotFoundError: No module named 'dask'
@aurghs
Copy link
Member

aurghs commented Apr 28, 2021

We could:

  1. Add can add a check to skip tests with dask if dask is not installed.
  2. Improve the message error message. That should be done in rioxarray

@alexamici
Copy link
Member

alexamici commented Apr 28, 2021

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

@aurghs I propose to do in the reverse order:

  1. open a PR to rioxarray where it raises an exception with a better error message
  2. (if 1. fails) add the same check in xarray-sentinel

@aurghs
Copy link
Member

aurghs commented Apr 28, 2021

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

1 and 2 are not an aut-aut. I would:

  • Add a check to skip tests (pyest) with dask if dask is not installed.
  • Add dask in setup.cfg as optional
  • open a PR to rioxarray where it raises an exception with a better error message

@alexamici
Copy link
Member

  1. we do install dask for testing already, no need to test without dask, we don't have any dask related logic to test. (I propose no action)
  2. dask is not a needed dependency so it must not be listed in the install_requires (no action)
  3. we agree, @aurghs would you please take care of it?

@aurghs
Copy link
Member

aurghs commented Apr 28, 2021

  1. we could add an xfails on the tests with dask
  2. I agree
  3. I think @corrado9999 was preparing the PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

3 participants