Process does not always exit on completion of data ingestion

[heffergm]

Node process continues to run and output to stderr, but by all appearances no counters are incremented in the log, leaving the impression that there's no data left to process.

Sample extract where I've seen this: https://s3.amazonaws.com/metro-extracts.mapzen.com/rome_italy.osm.pbf

[aberrios85]

+1

Seen same issue, stops pelias chef recipe completing.

Seen on imports of west-yorkshire and south-yorkshire osm.pbf GBR imports.

[cesozgen]

Any update or workaround to this?

[sevko]

@cesozgen , yes, a quick and very dirty hack: change the module.exports = stats; in /stream/stats.js to module.exports = through.obj;. This will prevent the OSM parser from spitting out ingestion stats (though you'll still see elasticsearch indexer stats), but the process will exit. Here's the reason why:

tl;dr, the OSM ingestion statistics-logging setInterval() isn't getting clearInterval()'d.

The stats module exports a stream that tracks the number of records that pass through, and stores the total number of such streams instantiated globally. The counter is incremented on pipe and decremented on unpipe events respectively: the interval gets set when the first pipe's created and torn down when the last one's closed, which is inferenced from the value of the counter. It turns out that the number of pipes and unpipes detected is !=, meaning that the counter doesn't drop to 0 like it should.

[missinglink]

resolved by #27