Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GTiff: improve performance of internal overview creation (fixes #1442) #1516

Merged
merged 1 commit into from
May 9, 2019

Conversation

rouault
Copy link
Member

@rouault rouault commented May 7, 2019

Or multiple level external overview.

This was a long standing problem. The issue was that each overview level (this
also applies to mask) within a same TIFF file shared the same TIFF* handle.
When switching between overviews, the handle had to be reset to point to a
new TIFF IFD, and reloading of the StripOffset/StripByteCount/TileOffset/TileByteCount
was done. For really big files, those can be several ten to hundred of MB large.

Now, each overview (mask) owns its own TIFF handle, sharing the underlying file
pointer though. Some care is needed in the tifvsi layer regarding some write
optimizations, as well when creating overviews, we must take care of forcing
a reload of the main TIFF handle so that the in-memory TIFF structure points
correctly to the next IFD.

Test case:

from osgeo import gdal
gdal.GetDriverByName('GTiff').Create('large.tif', 100000, 100000, 1, options = ['TILED=YES', 'SPARSE_OK=YES'])

With this improvement:
$ cp large.tif input.tif; time gdaladdo input.tif 2 4 8 --debug on
real 0m54.913s

Without:
real 3m0.175s

…#1442)

Or multiple level external overview.

This was a long standing problem. The issue was that each overview level (this
also applies to mask) within a same TIFF file shared the same TIFF* handle.
When switching between overviews, the handle had to be reset to point to a
new TIFF IFD, and reloading of the StripOffset/StripByteCount/TileOffset/TileByteCount
was done. For really big files, those can be several ten to hundred of MB large.

Now, each overview (mask) owns its own TIFF handle, sharing the underlying file
pointer though. Some care is needed in the tifvsi layer regarding some write
optimizations, as well when creating overviews, we must take care of forcing
a reload of the main TIFF handle so that the in-memory TIFF structure points
correctly to the next IFD.

Test case:
```
from osgeo import gdal
gdal.GetDriverByName('GTiff').Create('large.tif', 100000, 100000, 1, options = ['TILED=YES', 'SPARSE_OK=YES'])
```

With this improvement:
$ cp large.tif input.tif; time gdaladdo input.tif 2 4 8 --debug on
real	0m54.913s

Without:
real	3m0.175s
@rouault rouault merged commit ceb3e55 into OSGeo:master May 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant