Error: too many open files

[dphiffer]

Something odd happens when I try to generate concordances from https://github.com/whosonfirst-data/whosonfirst-data-postalcode-es

$ make concordances
grep: .gitattributes: No such file or directory
test -s /usr/local/data/whosonfirst-data-postalcode-es/bin/wof-build-concordances || make bin
mkdir -p /usr/local/data/whosonfirst-data-postalcode-es/meta
/usr/local/data/whosonfirst-data-postalcode-es/bin/wof-build-concordances -processes 1 -repo /usr/local/data/whosonfirst-data-postalcode-es
error: open /usr/local/data/whosonfirst-data-postalcode-es/data/554/832/469: too many open files
error: open /usr/local/data/whosonfirst-data-postalcode-es/data: too many open files

[thisisaaronland]

This is code that needs a throttle for open filehandles (that or something isn't closing its filehandles properly...)

[thisisaaronland]

Work in progress (in the throttle branch) but:

./bin/wof-concordances-list -mode directory /usr/local/data/whosonfirst-data/data
17:42:49.679310 [wof-concordances-list][index] STATUS time to index directory '/usr/local/data/whosonfirst-data/data' 7m8.20271976s
17:42:49.679380 [wof-concordances-list][index] STATUS time to index path '/usr/local/data/whosonfirst-data/data' 7m8.20281376s
17:42:49.679385 [wof-concordances-list][index] STATUS time to index paths (1) 7m8.202819701s
faa:code,inegi:id,icao,latimes:id,ne:sov_a3,nyt:id,id,hasc,marc:id,4sq:id,iata,qs:id,wk:page,tgn:id,gaul:id,wmo:id,uncrt:id,qs_pg:id,oa:id,fifa:id,fips:code,unlc:id,m49:code,itu:id,mzb:id,loc:id,sg:id,icao:code,fct:id,dbp:id,iso:id,iata:code,gn:id,gp:id,wd:id,fb:id,ioc:id,wikipedia,wikidata,hasc:id,woe:id,ne:adm0_a3

[thisisaaronland]

/usr/local/mapzen/go-whosonfirst-concordances/bin/wof-build-concordances
21:06:49.822595 [wof-build-concordances][index] STATUS time to index directory '/usr/local/data/whosonfirst-data/data' 7m36.481430822s
21:06:49.823893 [wof-build-concordances][index] STATUS time to index path '/usr/local/data/whosonfirst-data' 7m36.483274777s
21:06:49.824086 [wof-build-concordances][index] STATUS time to index paths (1) 7m36.483288595s
21:14:59.723679 [wof-build-concordances][index] STATUS time to index directory '/usr/local/data/whosonfirst-data/data' 8m9.888069042s
21:14:59.723693 [wof-build-concordances][index] STATUS time to index path '/usr/local/data/whosonfirst-data' 8m9.888107767s

wc -l meta/wof-concordances-latest.csv 
  620197 meta/wof-concordances-latest.csv

I am going to fix issue #5 as part of this ticket...

[thisisaaronland]

Should work now. I was able to generate concordances on a local laptop...

/usr/local/mapzen/go-whosonfirst-concordances/bin/wof-build-concordances
22:10:14.523610 [wof-build-concordances][index] STATUS time to index directory '/usr/local/data/whosonfirst-data/data' 6m55.427472911s
22:10:14.526101 [wof-build-concordances][index] STATUS time to index path '/usr/local/data/whosonfirst-data' 6m55.430806041s
22:10:14.526111 [wof-build-concordances][index] STATUS time to index paths (1) 6m55.430824218s

wc -l meta/wof-concordances-latest.csv
  620217 meta/wof-concordances-latest.csv

[dphiffer]

Confirmed. Thanks!