Smarter area weight grid saves memory in grdfilter #7192
Conversation
Instead of precomputing a weight grid for all input nodes, we only do so for 3 nodes along each row (west, middle, east) since all the columns away from the ends have the same weight along a row. It also speeds up the precalculation but mostly saves memory for large grids.
|
Another thought. Why does the second array (the 14 GB) needs to be allocated at all? Why not reuse the original array to store the filtered data? Filtered array is always small so we are safe on this side. If the grid was read from a disk file are sure to not destroy anything. The problem only poses if the grid was transmitted via a memory location, i.e. via wrappers. Then yes a new array is needed, but the other case no. |
|
Sorry, no good. When we are at output node IJ we need all input node within the radius. If you then treated the input via a separate node-ij system and placed the result there then at IJ+1 we now suddenly are averaging in a mix of input points and a previously placed output node. Only in very special cases will this not happen, and I think it must always happen at the beginning. Just think of that convolution circle slowly moving across the grid - it cannot pick up filtered values. |
|
Yes, that's right. I'm already imagining a complex schema where ... but too complex. |
Instead of precomputing an area weight grid for all input nodes (and hence doubling the memory requirement of the grid), we now only do so for 3 nodes along each row (west, middle, east) since all the columns away from the ends have the same weight along a row and only vary with latitude. The west and east columns may differ depending on boundary conditions, periodicity (360), and gridline vs pixel registration. It also speeds up the precalculation itself but mostly saves memory for large grids to be filtered in grdfilter. The case that triggered this was a 16 Gb in-memory grid that resulted in another 16 Gb memory grid being used - this now drops down to 1 Mb.
I have run all tests and it looks unchanged. Happy to have someone do a few filter operations to see if they notice anything, but the new logic looks solid to me. I had to add a silly API bool variable to bypass a test that for this strange grid (say 3 columns and 43600 rows) would fail in BC setting - this skips that check.