Importer chokes on nonexistent macrocounty entry

[cag]

Basically, the latest wof data has this issue:

whosonfirst-data/whosonfirst-data#543

and it causes the importer to complain:

2016-12-02T22:00:44.373Z - info: [whosonfirst] Loading county records from /home/dev/pelias-data/whosonfirst/meta/wof-county-latest.csv
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: ENOENT: no such file or directory, open '/home/dev/pelias-data/whosonfirst/data/110/849/885/6/1108498856.geojson'

[missinglink]

looks like this is due to the new id format, previously they were max 9 numbers, that one is 10 digits long.

[orangejulius]

I have some code somewhere to handle the new ID format, I'll submit the PR today.

[orangejulius]

Update: it appears that this is actually caused by an error in the WOF bundles. The whosonfirst importer uses the path from the meta files correctly. Here's a script that demonstrates the current issue:

#!/bin/bash -x

missing_id=1108498856

wget https://whosonfirst.mapzen.com/bundles/wof-macrocounty-latest.csv
grep $missing_id wof-macrocounty-latest.csv
wget https://whosonfirst.mapzen.com/bundles/wof-macrocounty-latest-bundle.tar.bz2
tar xf wof-macrocounty-latest-bundle.tar.bz2

find . -name "${missing_id}*"

[tylux]

@orangejulius is there a workaround to start importing the WOF bundles? Or just sit tight until the issue is resolved? Thanks!

[orangejulius]

Hey @tylux,
I'd recommend sitting for a bit, hopefully it will be fixed soon. If you really care about getting an import done more than data integrity, you can remove the line with that id in the wof-macrocounty-latest.csv file.

[orangejulius]

Hey @tylux and @cag,
The WOF bundle has been fixed, if you get a chance try the importer again, everything should work.

[tylux]

Hello @orangejulius seems to be fixed so far on the import.

EDIT: Now running into this error about 10 minutes into a full import with venues on a machine with 16GB mem and 8vcpus "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory" Not sure if related

[orangejulius]

Great, glad it's fixed. The WOF importer intentionally fails loudly on any issue reading files (we just made it even stricter in #178), so I'm closing this issue now. The memory leak is tracked in #180 and hopefully we can fix that soon.