Geocode fails with large timeseries file

[EJFielding]

Description of the problem
Geocoding step failed when it got to the timeseries.h5 file, with a "killed" message. I am guessing that it ran out of memory. My timeseries file has shape (87, 3988, 9791).

Full script that generated the error

smallbaselineApp.py RidgecrestSenA064.txt --dostep geocode

Full error message

******************** step - geocode ********************
create directory: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo
geocode.py -l /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/inputs/geometryRadar.h5 -t /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/smallbaselineApp.cfg --outdir /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo --update  /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/inputs/geometryRadar.h5 temporalCoherence.h5 timeseries.h5 velocity.h5
read input option from template file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/smallbaselineApp.cfg
number of processor to be used: 1
output pixel size in (lat, lon) in degree: (-0.00038907328792942296, 0.00031656142518274386)
output area extent in (S N W E) in degree: (35.002445220947266, 36.55368, -119.366615, -116.2674789428711)
--------------------------------------------------
resampling file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/inputs/geometryRadar.h5
reading azimuthAngle       from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading height             from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading incidenceAngle     from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading latitude           from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading longitude          from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading shadowMask         from geometryRadar.h5 ...
restrict fill value to False for bool type source data
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
reading slantRangeDistance from geometryRadar.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
create HDF5 file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_geometryRadar.h5 with w mode
create dataset /azimuthAngle       of float32    in size of (3988, 9791)         with compression=gzip
create dataset /height             of float32    in size of (3988, 9791)         with compression=gzip
create dataset /incidenceAngle     of float32    in size of (3988, 9791)         with compression=gzip
create dataset /latitude           of float32    in size of (3988, 9791)         with compression=gzip
create dataset /longitude          of float32    in size of (3988, 9791)         with compression=gzip
create dataset /shadowMask         of bool       in size of (3988, 9791)         with compression=gzip
create dataset /slantRangeDistance of float32    in size of (3988, 9791)         with compression=gzip
finished writing to /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_geometryRadar.h5
--------------------------------------------------
resampling file: temporalCoherence.h5
reading temporalCoherence from temporalCoherence.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
^A
update REF_LAT/LON/Y/X
create HDF5 file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_temporalCoherence.h5 with w mode
create dataset /temporalCoherence of float32    in size of (3988, 9791)         with compression=None
finished writing to /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_temporalCoherence.h5
--------------------------------------------------
resampling file: timeseries.h5
reading timeseries from timeseries.h5 ...
nearest resampling with kd_tree using 1 processor cores in 39 segments ...
Killed

System information

• Operating system: Linux
• Version of Python and relevant dependencies if applicable 3.7
• Version of MintPy (output of smallbaselineApp.py -v): MintPy release version v1.2beta-21, release date 2019-08-25

[hfattahi]

Yes probably memory issue. If you only need to geocode velocity file, then try to directly call geocode script:

geocode.py velocity.h5

[EJFielding]

Thanks @hfattahi, that works great for now.

[yunjunz]

This issue should be solved by the new geocode.py from #488 with block-by-block IO.

[EJFielding]

Thanks for returning to close this issue @yunjunz.

[EJFielding]

I went back to the dataset that was crashing before and ran the geocoding again. The geocoding of the large timeseries.h5 was successful!

grab dataset structure from ref_file: timeseries.h5
create HDF5 file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5 with w mode
create dataset  : bperp      of float32                   in size of (87,)                with compression = None
create dataset  : date       of |S8                       in size of (87,)                with compression = None
create dataset  : timeseries of float32                   in size of (87, 3988, 9791)     with compression = None
close  HDF5 file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 0, 307, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 307, 614, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 614, 921, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 921, 1228, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 1228, 1535, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 1535, 1842, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 1842, 2149, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 2149, 2456, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 2456, 2763, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 2763, 3070, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 3070, 3377, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 3377, 3684, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------
reading timeseries in block (0, 0, 9791, 3988) from timeseries.h5 ...
nearest resampling with pyresample.kd_tree using 1 CPU cores in 4 segments ...
write data in block [0, 87, 3684, 3988, 0, 9791] to file: /net/kraken/nobak/fielding/Ridgecrest/S1AB/A064/stack2/MintPy_v1/geo/geo_timeseries.h5
--------------------------------------------------

Thanks!

I should also say that I noticed that the smallbaselineApp.py script noticed that my input configuration file from August 2019 was old and automatically upgraded it, so I did not have to re-run the whole processing. Nice!