mapraster.c: problematic dereferencing of kerneldensity dataset

[gogglesguy]

Summary:
The in-memory kernel density heatmap gdal dataset handle (hDS) gets re-used but the associated rasterband cache doesn't get flushed, which causes old data to be rendered, instead of the newly generated imagery.

We've been testing the new Kernel Density Heatmap feature in mapserver 7.0. One major issue we've been seeing is that after running for a while mapserver starts returning incorrect heatmap data to our WMS requests. Eventually, it will be become pretty bad when just panning the map will return the exact same image back as the previous view. This made gave us the suspicion we were getting old cached data somehow.

This is has been easily reproducible with gdal < 2.0.
In gdal 2.0 and above I can't reproduce due to changes in the Memory driver that may have inadvertently fixed this issue in mapserver.

The situation:

msDrawRasterLayerLow calls msComputeKernelDensityDataset and receives a hDS handle. By default this handle only gets dereferenced ("DEFER" seems to be the default). After it is dereferenced, the memory attached to this dataset gets freed in msCleanupKernelDensityDataset. However the rasterband cache doesn't get flushed. This causes rendering issues later on.

This is technically just wrong. If you delete the underlying data, you can't just keep hanging on to the dataset handle.

msComputeKernelDensityDataset uses GDALOpenShared to return a new dataset. We've noticed that the longer it runs, the more likely a previous GDALDataSet is returned due to the fact that the memory will be allocated at the same location in memory for the same image size. See the following from one our logs where subsequent request used the same dataset:

MEM:::DATAPOINTER=0x82f48c0,PIXELS=1420,LINES=1098,BANDS=1,DATATYPE=Byte,PIXELOFFSET=1,LINEOFFSET=1420
MEM:::DATAPOINTER=0x82f48c0,PIXELS=1420,LINES=1098,BANDS=1,DATATYPE=Byte,PIXELOFFSET=1,LINEOFFSET=1420

Proposed solution is to always close the dataset handle in case of MS_KERNELDENSITY. It is the cleaner approach, especially since the underlying data does get deleted.

if (layer->connectiontype == MS_KERNELDENSITY) {
GDALClose(hDS);
}
else { /* previous dereference logic */ }

Alternatively, a call to GDALFlushCache(hDS) could help as well.

[tbonfort]

Thanks for the investigation and fix @gogglesguy . Applied to branch-7-0 in 3493927.

I was wondering if we shouldn't be explicitly setting the CLOSE_CONNECTION flag on the KD layer to avoid polluting mapraster.c with outside checks, but there are already many specific hooks in that file, so went with your fix as is.