Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Shape plugin keeps stale references to deleted files #784

Open
artemp opened this Issue · 10 comments

2 participants

@artemp
Owner

Mapnik (trunk) seems to keep shapefiles open once it finishes with them which causes stale data to be rendered if the file is deleted and replaced with another one with the same name in the meantime. lsof confirms it:

# After rendering isocronas.shp:
sigym@sigym-staging2:~$ md5sum /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
7649e466bc4a8ff01d370de569a1fa87  /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# set ISOCRONAS_INODE to the inode of this file retrieved with stat
sigym@sigym-staging2:~$ lsof | grep $ISOCRONAS_INODE
apache2    6614    sigym  mem       REG      202,6     68204 3132969133 /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# some process modifies isocronas.shp...
md5sum /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
285b592e4f0130750e99e8ba88057107  /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
sigym@sigym-staging2:~$ lsof | grep $ISOCRONAS_INODE
apache2    6614    sigym  DEL       REG      202,6           3132969133 /var/local/sigym/ctxs/propag_luis_k1c2dk/isocronas.shp
# notice how apache2 still refers to the old and deleted file

If apache is restarted then the correct shapefile is rendered. This regression was introduced somewhen after r2344 but I can't find the exact revision. Any clues?

@artemp
Owner

[albertov] I forgot to add that the Map, Datasource and every related instances are freshly created for every render by my app, they are not reused (which might have explained this behavior).

@artemp
Owner

[albertov] Configuring mapnik with {{{SHAPE_MEMORY_MAPPED_FILE=False}}} fixes (or works-around?) the issue. Perhaps the code-paths taken when setting this setting should take into account if the file's inode has changed? or perhaps make this a runtime configuration option instead of a compile-time option for shapefiles which are expected to be replaced?

@artemp
Owner

[artem] We need to check if mapping region is still valid when returning cached pointer. Couple ways to fix it, thanks for reporting this!

@artemp
Owner

[albertov] Can I help with closing this before 2.0.1? If no definite solution can be found perhaps {{{SHAPE_MEMORY_MAPPED_FILE}}} can be set to False by default before the release so packagers don't end up distributing versions with this bug.

Alberto

@artemp
Owner

[springmeyer] I've assumed that memory mapping being default on is pretty critical for performance, but have never actually tested. alberto, is this something you could do? See how much of a benefit the mapping gives for indexed shapefiles? If it does not have a big benefit then I might feel different about disabling this by default. Either way I figure it would be nice to have it as a runtime option eventually.

@artemp
Owner

[springmeyer] see also mapbox/tilemill#714 which may be related to the mapped regions being invalid - not sure...

@artemp
Owner

[springmeyer] alberto - also note that the issue may not be the memory mapping but that we cache the mapped regions. it would likely be pretty easy to add an option to the shapefile driver to be able to still use memory mapping, but just not to cache the regions.

@artemp
Owner

[artem] Moving to 2.0.1

@springmeyer
Owner

@albertov - hey there, is this still problematic for you? Take a look at #1022 - would exposing the clear() method in python be enough for you?

@springmeyer
Owner

moving to 2.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.