Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory-mapped raster backend #37

Open
6 tasks
fb71 opened this issue Apr 9, 2017 · 0 comments
Open
6 tasks

Memory-mapped raster backend #37

fb71 opened this issue Apr 9, 2017 · 0 comments

Comments

@fb71
Copy link
Contributor

fb71 commented Apr 9, 2017

Efficiently handle rasters that exceed heap and main memory size.

Current implementation ueses default DataBuffer backend for WriteableRaster as backend for GridCoverage. DataBuffer uses a plan primitive[] array for data storage. That is, a raster can have max. 2G = 46K x 46K raster elements and allocates 2GB x 8 = 16GB size on heap.

Goals

  • process raster more independently of actual memory and heap size
  • allow to process rasters > 2GB on 32bit systems
  • maybe use memory-mapped files as persistent storage, avoid write/encode after processing

TODO

  • eviction: limit mapped files on 32bit systems
  • memory sensitive on 32bit systems (soft references and check for allocation error)
  • check whole-file version for 64bit systems
  • stride/tile version
  • check memory-mapped files as persistent storage
  • check double/float/int data type for modules
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant