How to reduce the memory demands of AP4T

[dt-woods]

1. Use file handlers to HDF5
   • For example, Cnty_MC is read from HDF5 file stream, S.
   • Check to see if Cnty_MC values ever change. If not, then we could write a method every time we want a value from Cnty_MC, send a method S, i, and j which returns one matrix at at time.
   • Can we create HDF5 to hold IDW distribution data?
   • Can we store results in an HDF5 file (to avoid having it in memory)? Save as last resort.
2. Break up the process

[dt-woods]

New approach to Cnty_MC, which currently depends on 'cnty_mc.h5'. Because h5read is not supported in Octave, we are left with the load function that reads all the data into memory. To avoid the memory burden, let's break apart the {3,5} cell into 15 HDF5 files.

[dt-woods]

Better yet! The package hdf5oct package provides the h5read method that does not pull the full HDF5 into memory! Here's how:

• Requires: hdf5oct package (https://gapost.github.io/hdf5oct/)
• Installation instructions (https://gnu-octave.github.io/packages/hdf5oct/)

Here's a method

Tract_to_Tract_Calibration.m

Cnty_MCs = Cnty_MC{1,3}(idx,:);
% THE NEW WAY: Read matrix from function that pulls matrix from file
Cnty_MCs = read_county_hdf(1, 3)(idx, :);
% Option 1:
Cnty_MCs = h5read([hdf_dir "cnty_mc.h5"], "/3/3")(idx, :)
% Option 2:
col_size = size(h5read([hdf_dir "cnty_mc.h5"], "/3/3"), 2);
Cnty_MCs = h5read([hdf_dir "cnty_mc.h5"], "/3/3", [idx, 1], [1, col_size])

[dt-woods]

The start vector tells h5read the row and column indices to start with. Here it's the row, idx (the current EGU row), and column 1 (always the first column). The count vector tells h5read how many rows and how many columns to read. Here, it's one row (the current idx only) and all columns (which is defined by col_size).