Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loading cache for resampling scene fails with numpy 1.16.3 #785

Closed
observingClouds opened this issue May 20, 2019 · 3 comments · Fixed by #787 · May be fixed by observingClouds/satpy#1
Closed

Loading cache for resampling scene fails with numpy 1.16.3 #785

observingClouds opened this issue May 20, 2019 · 3 comments · Fixed by #787 · May be fixed by observingClouds/satpy#1

Comments

@observingClouds
Copy link
Contributor

Describe the bug
With the new release of Numpy (1.16.3) the

unpickling object arrays
when loading *.npy or *.npz files now requires an explicit opt-in.
This backwards incompatible change was made in response to
CVE-2019-6446 https://nvd.nist.gov/vuln/detail/CVE-2019-6446_.

(https://github.com/numpy/numpy/releases)

Due to this change the function call scene.resample() fails when cache_dir is given and used to load an existing cache.

To Reproduce

from satpy import Scene

datafile = SomeGOES16File.nc
scn = Scene(reader="abi_l1b", filenames=[datafile])
scn.load(['C13'])
new_scn = scn.resample(scn.min_area(), cache_dir='./') #Writing cache

new_scn2 = scn.resample(scn.min_area(), cache_dir='./') #Using cache

Expected behavior
new_scn should include resampled scene when using cache_dir and resample cache exists

Actual results

new_scn = scn.resample(scn.min_area(), cache_dir='./')
  File "../python3.6/site-packages/satpy/scene.py", line 1101, in resample
    reduce_data=reduce_data, **resample_kwargs)
  File "../python3.6/site-packages/satpy/scene.py", line 1056, in _resampled_scene
    res = resample_dataset(dataset, destination_area, **kwargs)
  File "../python3.6/site-packages/satpy/resample.py", line 916, in resample_dataset
    new_data = resample(source_area, dataset, destination_area, fill_value=fill_value, **kwargs)
  File "../python3.6/site-packages/satpy/resample.py", line 879, in resample
    res = resampler_instance.resample(data, **kwargs)
  File "../python3.6/site-packages/satpy/resample.py", line 271, in resample
    cache_id = self.precompute(cache_dir=cache_dir, **kwargs)
  File "../python3.6/site-packages/satpy/resample.py", line 352, in precompute
    self.load_neighbour_info(cache_dir, mask=mask, **kwargs)
  File "../python3.6/site-packages/satpy/resample.py", line 383, in load_neighbour_info
    new_cache = dict(cache.items())
  File "../python3.6/_collections_abc.py", line 744, in __iter__
    yield (key, self._mapping[key])
  File "../python3.6/site-packages/numpy/lib/npyio.py", line 262, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "../python3.6/site-packages/numpy/lib/format.py", line 692, in read_array
    raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False

Environment Info:

  • OS: Linux
  • Satpy Version: 0.15.1
  • PyResample Version: 1.12.3
@observingClouds
Copy link
Contributor Author

Possible solution
Change the following line 381 in satpy/resample.py from
cache = np.load(filename, mmap_mode='r')
to
cache = np.load(filename, mmap_mode='r', allow_pickle=True)

@pnuu
Copy link
Member

pnuu commented May 20, 2019

Hmm. Maybe we should finally have a look at using zarr for the caches. @djhoese what do you think?

@djhoese
Copy link
Member

djhoese commented May 21, 2019

@pnuu Yes, but we should include @observingClouds fix for the next bug release too. I've asked @observingClouds to resubmit their PR to this repository (currently only on their own fork).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants