Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make hdfeos_l1b geo interpolation use dask arrays #370

Closed
wroberts4 opened this issue Jul 16, 2018 · 4 comments
Closed

Make hdfeos_l1b geo interpolation use dask arrays #370

wroberts4 opened this issue Jul 16, 2018 · 4 comments
Labels

Comments

@wroberts4
Copy link
Contributor

wroberts4 commented Jul 16, 2018

Is your feature request related to a problem? Please describe.
_interpolate under satpy.readers.hdfeos_l1b could be more efficient if arrays were converted to be lazy.

Describe the solution you'd like
Convert numpy arrays to dask arrays. This will allow the arrays to be "lazy".

Code to be improved

def _interpolate(data, coarse_resolution, resolution, interpolator=None):
if resolution == coarse_resolution:
return data
if interpolator is None:
from geotiepoints.interpolator import Interpolator
interpolator = Interpolator
logger.debug("Interpolating from " + str(coarse_resolution)
+ " to " + str(resolution))
if isinstance(data, (tuple, list, set)):
lines = data[0].shape[0]
else:
lines = data.shape[0]
if coarse_resolution == 5000:
coarse_cols = np.arange(2, 1354, 5)
lines *= 5
coarse_rows = np.arange(2, lines, 5)
elif coarse_resolution == 1000:
coarse_cols = np.arange(1354)
coarse_rows = np.arange(lines)
if resolution == 1000:
fine_cols = np.arange(1354)
fine_rows = np.arange(lines)
chunk_size = 10
elif resolution == 500:
fine_cols = np.arange(1354 * 2) / 2.0
fine_rows = (np.arange(lines * 2) - 0.5) / 2.0
chunk_size = 20
elif resolution == 250:
fine_cols = np.arange(1354 * 4) / 4.0
fine_rows = (np.arange(lines * 4) - 1.5) / 4.0
chunk_size = 40
along_track_order = 1
cross_track_order = 3
satint = interpolator(data,
(coarse_rows, coarse_cols),
(fine_rows, fine_cols),
along_track_order,
cross_track_order,
chunk_size=chunk_size)
satint.fill_borders("y", "x")
return satint.interpolate()

Additional context
@djhoese suggested this improvement.

@djhoese djhoese added enhancement code enhancements, features, improvements help wanted component:readers optimization labels Jul 16, 2018
@djhoese
Copy link
Member

djhoese commented Jul 16, 2018

@wroberts4 your issue doesn't reference what reader this is for. It would also be helpful if you could point to the specific range of lines or the file where this interpolation is started. Please update the issue to include this information including modifying the title. Thanks.

@wroberts4 wroberts4 changed the title geolocation interpolation optimization Make _interpolate use dask arrays Jul 16, 2018
@wroberts4 wroberts4 changed the title Make _interpolate use dask arrays Make hdfeos_l1b geo interpolation use dask arrays Jul 16, 2018
@djhoese
Copy link
Member

djhoese commented Nov 26, 2018

@mraspaud You fixed this in 0.10 (can't find the PR right now) didn't you?

@mraspaud
Copy link
Member

Well, I didn't fix that code, but switch to the new interpolation we implemented in #430 for Modis makes it dask and xarray friendly.

@mraspaud
Copy link
Member

This has been implemented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants