Exiting with unfinished jobs

[kldavis4]

I wrote an app which parses a large xml file with around 17M items of interest. These are added to a queue and using batch processing, written to a web service. When parsing is complete, the app exits (status code 0) with about half of the items are left unwritten. Any idea what might be going on?

[leanderlee]

Node will exit if it does not have any callbacks waiting. Try hooking an event to drain or put a callback that waits for the last event to finish to prevent the process from exiting.

[kldavis4]

Awesome, thanks for your help.

[kldavis4]

I attached a listener for the drain event and it is firing only once, just before exiting. There are still 11.5M tasks remaining at that point. What is the best way to see how many remaining tasks are queued? My intuition is that tasks are getting lost somehow, as I am not using a persistent store, and I would expect memory issues with that many queued tasks.

[leanderlee]

You can try calling queue.getStats() to see the total. It would be a bug if it calls drain before it's completed all of the tasks unless tasks were getting completed faster than they are being pushed on. Do you have some code to reproduce this?

[kldavis4]

Yeah, I am calling getStats on that drain event, but that is telling me how many have been completed, not how many are remaining, right? Any tips on getting better visibility into queue state would help a lot.

I do have code which I can share, but it currently involves parsing the english wikipedia article dump and posting it to elasticsearch. I can try to reduce it to something a bit more manageable, but it will take some time.

[leanderlee]

One way you can get more visibility is by passing in your own store (or modifying the memory store) and adding a counter. You really only need to implement 5 functions or stub them out to see if it's working properly.

https://github.com/diamondio/better-queue#custom-store

[kldavis4]

Ok, great. Let me give that a go and I'll report what I find.

[kldavis4]

Ok, mystery solved.. My task object had a non-unique id field derived from my input and I didn't explicitly configure a unique id so a large number of tasks were getting merged. Thanks for your help in troubleshooting.