Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

r.external speed-up #1305

Merged
merged 5 commits into from
Feb 7, 2021
Merged

r.external speed-up #1305

merged 5 commits into from
Feb 7, 2021

Conversation

metzm
Copy link
Contributor

@metzm metzm commented Feb 4, 2021

Use GDALComputeRasterMinMax() for faster range computation.
Add super-fast mode without range computation, and a warning for this mode.

r.info will not report the range if not existing. If the range is explicitly requested with -r or -s, the range and stats are computed on-the-fly if not existing.

r.support -s is now computing the correct range for FCELL /DCELL maps.

Tested with a tiled GeoTIFF with 7 billion cells, linking with r.external drops from 2:06 minutes to 46 seconds. In super-fast mode, linking takes only 0.05 seconds.

Regarding the super-fast mode without data range computation, a number of modules will not work correctly. If the range is needed, it can be computed for the linked map with r.support, but linking again with r.external would be faster, at least for large, tiled maps.

 * faster linking with r.external
 * super-fast mode for r.external
 * update r.info if range is missing
 * fix r.support to re-compute range for fp maps
 * warning for fast mode
@metzm metzm added bug Something isn't working enhancement New feature or request labels Feb 4, 2021
@metzm metzm added this to the 8.0.0 milestone Feb 4, 2021
@metzm metzm requested review from neteler and landam February 4, 2021 17:00
lib/raster/range.c Outdated Show resolved Hide resolved
lib/raster/range.c Outdated Show resolved Hide resolved
@metzm
Copy link
Contributor Author

metzm commented Feb 5, 2021

A case for the super-fast mode is raster reprojection with r.proj. That means r.import can in most cases be faster, if reprojection is needed, if r.in.gdal is replaced with r.external -r in r.import. However, there must be a reason why r.import uses r.in.gdal instead of r.external when the input raster needs to be reprojected. IIRC, there were some problems with r.external that did not occur with r.in.gdal.

metzm and others added 3 commits February 5, 2021 21:46
Co-authored-by: Markus Neteler <neteler@gmail.com>
Co-authored-by: Markus Neteler <neteler@gmail.com>
 * add -s flag to the warning to use r.suport
@neteler
Copy link
Member

neteler commented Feb 7, 2021

However, there must be a reason why r.import uses r.in.gdal instead of r.external when the input raster needs to be reprojected. IIRC, there were some problems with r.external that did not occur with r.in.gdal.

I traced the snippet back to GRASS GIS addons, r.in.proj (commit from Jan 27, 2015):
OSGeo/grass-addons@59b6976#diff-d06b709384f210f1f01d068364921d06ba692c21040323997220dad9e1d9d0beR182

Already back then it was based on r.in.gdal.

(For those here not knowing the story: the r.in.proj addon was in OSGeo/grass-addons@d26fd4d renamed to r.import and later moved to GRASS-core)

@metzm
Copy link
Contributor Author

metzm commented Feb 7, 2021

However, there must be a reason why r.import uses r.in.gdal instead of r.external when the input raster needs to be reprojected. IIRC, there were some problems with r.external that did not occur with r.in.gdal.

I traced the snippet back to GRASS GIS addons, r.in.proj (commit from Jan 27, 2015):
OSGeo/grass-addons@59b6976#diff-d06b709384f210f1f01d068364921d06ba692c21040323997220dad9e1d9d0beR182

Already back then it was based on r.in.gdal.

r.import uses some options and flags with r.in.gdal that are not available in r.external, thus in r.import, r.in.gdal can not be simply replaced with r.external when it comes to the actual import of the data. Worth a separate issue/PR.

@metzm metzm merged commit 829734c into OSGeo:master Feb 7, 2021
@metzm metzm deleted the r_external_fast branch February 7, 2021 21:48
@neteler
Copy link
Member

neteler commented Feb 8, 2021

Due to the containing bugfix I suggest to backport this after a while

neteler added a commit that referenced this pull request Feb 18, 2021
 * faster linking with r.external
 * super-fast mode for r.external
 * warning for fast mode
 * update r.info if range is missing
 * fix r.support to re-compute range for fp maps

Co-authored-by: Markus Neteler <neteler@gmail.com>
@neteler neteler modified the milestones: 8.0.0, 7.8.6 Feb 18, 2021
marisn pushed a commit to marisn/grass that referenced this pull request Mar 22, 2021
 * faster linking with r.external
 * super-fast mode for r.external
 * warning for fast mode
 * update r.info if range is missing
 * fix r.support to re-compute range for fp maps

Co-authored-by: Markus Neteler <neteler@gmail.com>
ninsbl pushed a commit to ninsbl/grass that referenced this pull request Oct 26, 2022
 * faster linking with r.external
 * super-fast mode for r.external
 * warning for fast mode
 * update r.info if range is missing
 * fix r.support to re-compute range for fp maps

Co-authored-by: Markus Neteler <neteler@gmail.com>
ninsbl pushed a commit to ninsbl/grass that referenced this pull request Feb 17, 2023
 * faster linking with r.external
 * super-fast mode for r.external
 * warning for fast mode
 * update r.info if range is missing
 * fix r.support to re-compute range for fp maps

Co-authored-by: Markus Neteler <neteler@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants