Memory issues for chunks in global-files

[valpesendorfer]

Traceback:

Error message:

Traceback (most recent call last):
  File "C:\Python27\ArcGIS10.4\Lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\ArcGIS10.4\Lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "T:\VALENTIN\py27\wsmtk\Scripts\processMODIS.exe\__main__.py", line 9, in <module>
  File "t:\valentin\py27\wsmtk\lib\site-packages\wsmtk\processMODIS.py", line 82, in main
    h5.update()
  File "t:\valentin\py27\wsmtk\lib\site-packages\wsmtk\modis.py", line 263, in update
    dset[blk[0]:(blk[0]+self.chunks[0]),:,uix:(uix+arr.shape[2])] = arr[...]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "t:\valentin\py27\wsmtk\lib\site-packages\h5py\_hl\dataset.py", line 632, in __setitem__
    self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5d.pyx", line 221, in h5py.h5d.DatasetID.write
  File "h5py\_proxy.pyx", line 132, in h5py._proxy.dset_rw
  File "h5py\_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
IOError: Can't prepare for writing data (memory allocation failed for raw data chunk)

[valpesendorfer]

Couldn't be replicated ...

Might be due to corrupt files, PR #14 implements enhanced handling of corrupt files ... might be resolving the issue.

[valpesendorfer]

Turns out error is due to numpy running out of memory addresses using a PYTHON 32bit interpreter (even though there's plenty of empty RAM available)

See for reference: numpy/numpy#9960

PR #15 addresses this issue:

• deletion of np array for block after writing to h5 file seems to mitigate the problem (working for now, but might reapear - maybe depending on what else is in RAM)
• Adding a try-except section to catch memory issues and notify user

Maybe it should be added to the package description that it's to be run preferably with a PY 64 version.