Don't import "historic" WOF neighbourhoods

[nvkelso]

When a WOF record has either the edtf:cessation or the edtf:deprecated, we should treat it the same as a record that has been superseded_by and not import it for tile generation (and if it's been imported in the past, we should remove it).

[nvkelso]

See more:

• import: https://github.com/mapzen/tilequeue/blob/master/tilequeue/wof.py#L70-L72
• processing: https://github.com/mapzen/tilequeue/blob/master/tilequeue/wof.py#L161-L168
• purging: https://github.com/mapzen/tilequeue/blob/master/tilequeue/wof.py#L895-L899

[nvkelso]

/cc @thisisaaronland

[nvkelso]

Turns out these aren't in unix seconds but in YYYY-MM-dd formatted strings. So technically we should look at the dates and evaluate if they are in the future, and if they are, still allow them in today. But if they are today or yesterday or yesterday yesterday then drop them.

See also:

• https://github.com/whosonfirst/whosonfirst-dates

[zerebubuth]

I had a crack at this and it seems it's harder than I expected. Basically, we've assumed in the code (all the way through, not just the WOF import code) that visibility of a feature is a function of its parameters and geometry only. Supporting the inception, cessation and deprecated parameters means that the visibility of a feature is a function of the current time as well.

This isn't a show-stopper, just means that we have to carry through the start and end dates from all neighbourhoods through to the database (which might involve some interesting ways of encoding "year infinity" in the database) and index them and have the tile update tool be more clever about what tiles it considers for updates (it needs to have an idea of the "last time I ran").

Feel free to move it around in the priority list if this change in effort level changes the priority.

[nvkelso]

LGTM, let's try it on dev.

[nvkelso]

Hmm, I'm still seeing the bunk United States neighbourhood label in Mesa, Arizona:

https://whosonfirst.mapzen.com/spelunker/id/85865451/
/#15/33.3108/-111.6676

[zerebubuth]

Yeah, needed an update to the mapnik server as well. Once that was done, I forcibly refreshed that one tile: 15/6219/13165 (got tired of waiting for it in the humongous dev queue), and now it's gone. It's possible that there are others, but these should be caught when they're next updated.

[nvkelso]

I still see it in this tiles (and others)?

• http://vector.dev.mapzen.com/osm/all/15/6219/13165.topojson

Caught when they're next updates meaning any of that deprecated records other fields are updated or? That may be a while... does this need a DB migration that effectively reloads all the neighbourhoods?

[zerebubuth]

That tile might be cached already. The update that I did was pretty hacky, perhaps it can only be seen in the original json: http://vector.dev.mapzen.com/osm/all/15/6219/13165.json

We could do a mass-update, but there are so few neighbourhoods which have an inception / cessation that it's probably best to target some sort of one-off update for those.

[nvkelso]

Testing this with a new one that's marked deprecated today ("edtf:deprecated":"2016-01-21"):

• data/858/288/13/85828813.geojson

Which is Korblex in Arcata, California:

• abstract out SQS and S3 #15/40.9048/-124.0804

[nvkelso]

Worked.