Error reading SEVIRI native file from EUMETSAT API #1440
Labels
Comments
|
@cherif2019-dev thanks for raising this issue. |
|
Thank you Martin,
This is the code :
filenames = glob('../data/nat/*.nat')
scn = Scene(reader='seviri_l1b_native',filenames=filenames)
scn.load(['HRV'])
scn.save_datasets(filename='mydata.nc', engine='netcdf4')
ERROR :
…---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-8a25d3efcbdd> in <module>
1 scn.load(['HRV'])
2
----> 3 scn.save_datasets(filename='my_abi_data.nc', engine='netcdf4')
4
~\Anaconda3\envs\satpy\lib\site-packages\satpy\scene.py in save_datasets(self, writer, filename, datasets, compute, **kwargs)
1366 filename=filename,
1367 **kwargs)
-> 1368 return writer.save_datasets(datasets, compute=compute, **save_kwargs)
1369
1370 @staticmethod
~\Anaconda3\envs\satpy\lib\site-packages\satpy\writers\cf_writer.py in save_datasets(self, datasets, filename, groups, header_attrs, engine, epoch, flatten_attrs, exclude_attrs, include_lonlats, pretty, compression, **to_netcdf_kwargs)
702 datas, start_times, end_times = self._collect_datasets(
703 group_datasets, epoch=epoch, flatten_attrs=flatten_attrs, exclude_attrs=exclude_attrs,
--> 704 include_lonlats=include_lonlats, pretty=pretty, compression=compression)
705 dataset = xr.Dataset(datas)
706 if 'time' in dataset:
~\Anaconda3\envs\satpy\lib\site-packages\satpy\writers\cf_writer.py in _collect_datasets(self, datasets, epoch, flatten_attrs, exclude_attrs, include_lonlats, pretty, compression)
593 ds = ds.copy(deep=True)
594 try:
--> 595 new_datasets = area2cf(ds, strict=include_lonlats, got_lonlats=got_lonlats)
596 except KeyError:
597 new_datasets = [ds]
~\Anaconda3\envs\satpy\lib\site-packages\satpy\writers\cf_writer.py in area2cf(dataarray, strict, got_lonlats)
198 res = []
199 if not got_lonlats and (isinstance(dataarray.attrs['area'], SwathDefinition) or strict):
--> 200 dataarray = area2lonlat(dataarray)
201 if isinstance(dataarray.attrs['area'], AreaDefinition):
202 dataarray, gmapping = area2gridmapping(dataarray)
~\Anaconda3\envs\satpy\lib\site-packages\satpy\writers\cf_writer.py in area2lonlat(dataarray)
171 ignore_dims = {dim: 0 for dim in dataarray.dims if dim not in ['x', 'y']}
172 chunks = getattr(dataarray.isel(**ignore_dims), 'chunks', None)
--> 173 lons, lats = area.get_lonlats(chunks=chunks)
174 dataarray['longitude'] = xr.DataArray(lons, dims=['y', 'x'],
175 attrs={'name': "longitude",
~\Anaconda3\envs\satpy\lib\site-packages\pyresample\geometry.py in get_lonlats(self, nprocs, data_slice, cache, dtype, chunks)
2342 row_slice.step)
2343 lons, lats = definition.get_lonlats(nprocs=nprocs, data_slice=(local_row_slice, col_slice),
-> 2344 cache=cache, dtype=dtype, chunks=chunks)
2345
2346 llons.append(lons)
~\Anaconda3\envs\satpy\lib\site-packages\pyresample\geometry.py in get_lonlats(self, nprocs, data_slice, cache, dtype, chunks)
1941
1942 # Get X/Y coordinates for the whole area
-> 1943 target_x, target_y = self.get_proj_coords(data_slice=data_slice, chunks=chunks, dtype=dtype)
1944 if nprocs is None and not hasattr(target_x, 'chunks'):
1945 nprocs = self.nprocs
~\Anaconda3\envs\satpy\lib\site-packages\pyresample\geometry.py in get_proj_coords(self, data_slice, dtype, chunks)
1844
1845 """
-> 1846 target_x, target_y = self._get_proj_vectors(dtype=dtype, check_rotation=False, chunks=chunks)
1847 if data_slice is not None and isinstance(data_slice, slice):
1848 target_y = target_y[data_slice]
~\Anaconda3\envs\satpy\lib\site-packages\pyresample\geometry.py in _get_proj_vectors(self, dtype, check_rotation, chunks)
1787
1788 target_x = arange(self.width, **x_kwargs) * self.pixel_size_x + self.pixel_upper_left[0]
-> 1789 target_y = arange(self.height, **y_kwargs) * -self.pixel_size_y + self.pixel_upper_left[1]
1790 return target_x, target_y
1791
~\Anaconda3\envs\satpy\lib\site-packages\dask\array\creation.py in arange(*args, **kwargs)
361 dtype = np.arange(start, stop, step * num if num else step).dtype
362
--> 363 chunks = normalize_chunks(chunks, (num,), dtype=dtype)
364
365 if kwargs:
~\Anaconda3\envs\satpy\lib\site-packages\dask\array\core.py in normalize_chunks(chunks, shape, limit, dtype, previous_chunks)
2587 raise ValueError(
2588 "Chunks do not add up to shape. "
-> 2589 "Got chunks=%s, shape=%s" % (chunks, shape)
2590 )
2591
ValueError: Chunks do not add up to shape. Got chunks=((3711, 3711, 3714),), shape=(8640,)
Kind Regards,
Cherif
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: Martin Raspaud<mailto:notifications@github.com>
Sent: Tuesday, November 17, 2020 12:21 AM
To: pytroll/satpy<mailto:satpy@noreply.github.com>
Cc: cherif2019-dev<mailto:charfi008@hotmail.fr>; Mention<mailto:mention@noreply.github.com>
Subject: Re: [pytroll/satpy] /pytroll/satpy/issues (#1440)
@cherif2019-dev<https://github.com/cherif2019-dev> thanks for raising this issue.
We need more info in order to be able to help you.
Could you provide the smallest possible script that produces this error, and make the data file available for us to test if needed?
Thanks
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1440 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANUYARCMRB7CE6WN3QXXJVDSQGCSTANCNFSM4TXSYINQ>.
|
|
@cherif2019-dev |
|
This is the file name downloaded from eumetsat api (265MB):
MSG1-SEVI-MSG15-0100-NA-20190409201240.419000000Z-NA.nat
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: Martin Raspaud<mailto:notifications@github.com>
Sent: Tuesday, November 17, 2020 12:59 AM
To: pytroll/satpy<mailto:satpy@noreply.github.com>
Cc: cherif2019-dev<mailto:charfi008@hotmail.fr>; Mention<mailto:mention@noreply.github.com>
Subject: Re: [pytroll/satpy] /pytroll/satpy/issues (#1440)
@cherif2019-dev<https://github.com/cherif2019-dev>
the code looks good, could you share your data file?
@strandgren<https://github.com/strandgren> @ameraner<https://github.com/ameraner> do you have an idea?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1440 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANUYARFPXQLXJDCOCCUPSFTSQGHERANCNFSM4TXSYINQ>.
|
djhoese
changed the title
|
@cherif2019-dev @mraspaud It seems to be related to the fact that the native HRV data are not padded to full disk when loaded. I'm currently working on this and hope to have it finished in a few days. I did a quick test with my code in development and the lines of code from @cherif2019-dev seem to work fine with that nat-file. |
|
@strandgren thanks for checking! @cherif2019-dev the PR in question: #1438 |
5 tasks
|
Can confirm that the error occurs for me with the current satpy release. |
|
@cherif2019-dev if you need to have this working on a short notice you could clone my branch from the draft PR above. I am still working on improving the code in terms of memory usage, but the code itself code should do the job I think |
3 tasks
|
Hi all, this feels like a deja-vu, and indeed it's a bug in pyresample I discussed this summer with @djhoese but never got to fix. I just created a pyresample PR pytroll/pyresample#313 that fixes the problem. @strandgren 's satpy PR circumvents the issue since the output Dataset after padding has an |
|
@cherif2019-dev the mentioned pyresample PR has been merged, so if you use pyresample's master branch it should fix your issue. |
|
Thank you Martin, Please how can I use pyresample master branch?
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: Martin Raspaud<mailto:notifications@github.com>
Sent: Wednesday, November 18, 2020 9:20 PM
To: pytroll/satpy<mailto:satpy@noreply.github.com>
Cc: cherif2019-dev<mailto:charfi008@hotmail.fr>; Mention<mailto:mention@noreply.github.com>
Subject: Re: [pytroll/satpy] Error reading SEVIRI native file from EUMETSAT API (#1440)
@cherif2019-dev<https://github.com/cherif2019-dev> the mentioned pyresample PR has been merged, so if you use pyresample's master branch it should fix your issue.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1440 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANUYARCOIH4HT7F3QK6OEWLSQP64NANCNFSM4TXSYINQ>.
|
|
@cherif2019-dev you can clone the current master into your machine, then install it using pip: git clone git@github.com:pytroll/pyresample.git
pip install -e ./pyresample/ |
|
Thank you so much Andrea!!!
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
From: Andrea Meraner<mailto:notifications@github.com>
Sent: Wednesday, November 18, 2020 9:47 PM
To: pytroll/satpy<mailto:satpy@noreply.github.com>
Cc: cherif2019-dev<mailto:charfi008@hotmail.fr>; Mention<mailto:mention@noreply.github.com>
Subject: Re: [pytroll/satpy] Error reading SEVIRI native file from EUMETSAT API (#1440)
@cherif2019-dev<https://github.com/cherif2019-dev> you can clone the current master into your machine, then install it using pip:
git clone git@github.com:pytroll/pyresample.git
pip install -e ./pyresample/
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1440 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANUYARDRGP3A6GJHH5U6Z2LSQQCDHANCNFSM4TXSYINQ>.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I downloaded a seviri file from eumetsat api in nat format but when I tried to save the HRV channel in netcdf format I got a satpy error :
ValueError: Chunks do not add up to shape. Got chunks=((3711, 3711, 3714),), shape=(8640,)
Please do you have any idea about this error, Thank you.
The text was updated successfully, but these errors were encountered: