Shape plugin keeps stale references to deleted files

[artemp]

Mapnik (trunk) seems to keep shapefiles open once it finishes with them which causes stale data to be rendered if the file is deleted and replaced with another one with the same name in the meantime. lsof confirms it:

# After rendering isocronas.shp:
sigym@sigym-staging2:~$ md5sum /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
7649e466bc4a8ff01d370de569a1fa87  /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# set ISOCRONAS_INODE to the inode of this file retrieved with stat
sigym@sigym-staging2:~$ lsof | grep $ISOCRONAS_INODE
apache2    6614    sigym  mem       REG      202,6     68204 3132969133 /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# some process modifies isocronas.shp...
md5sum /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
285b592e4f0130750e99e8ba88057107  /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
sigym@sigym-staging2:~$ lsof | grep $ISOCRONAS_INODE
apache2    6614    sigym  DEL       REG      202,6           3132969133 /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# notice how apache2 still refers to the old and deleted file

If apache is restarted then the correct shapefile is rendered. This regression was introduced somewhen after r2344 but I can't find the exact revision. Any clues?

[artemp]

[albertov] I forgot to add that the Map, Datasource and every related instances are freshly created for every render by my app, they are not reused (which might have explained this behavior).

[artemp]

[albertov] Configuring mapnik with {{{SHAPE_MEMORY_MAPPED_FILE=False}}} fixes (or works-around?) the issue. Perhaps the code-paths taken when setting this setting should take into account if the file's inode has changed? or perhaps make this a runtime configuration option instead of a compile-time option for shapefiles which are expected to be replaced?

[artemp]

[artem] We need to check if mapping region is still valid when returning cached pointer. Couple ways to fix it, thanks for reporting this!

[artemp]

[albertov] Can I help with closing this before 2.0.1? If no definite solution can be found perhaps {{{SHAPE_MEMORY_MAPPED_FILE}}} can be set to False by default before the release so packagers don't end up distributing versions with this bug.

Alberto

[artemp]

[springmeyer] I've assumed that memory mapping being default on is pretty critical for performance, but have never actually tested. alberto, is this something you could do? See how much of a benefit the mapping gives for indexed shapefiles? If it does not have a big benefit then I might feel different about disabling this by default. Either way I figure it would be nice to have it as a runtime option eventually.

[artemp]

[springmeyer] see also https://github.com/mapbox/tilemill/issues/714 which may be related to the mapped regions being invalid - not sure...

[artemp]

[springmeyer] alberto - also note that the issue may not be the memory mapping but that we cache the mapped regions. it would likely be pretty easy to add an option to the shapefile driver to be able to still use memory mapping, but just not to cache the regions.

[artemp]

[artem] Moving to 2.0.1

[springmeyer]

@albertov - hey there, is this still problematic for you? Take a look at #1022 - would exposing the clear() method in python be enough for you?

[springmeyer]

moving to 2.1

[pessimo]

sorry to be here. I just try to use clear_cache() in python to apply some tiny modify in shapefile. however it seems not working. Is it already a function in python? I'm using version 3.0.20