Stalled Jobs

[jwarzech]

I have two instances of go-workers running on a couple vps servers (1GB ram -> 25 workers) and (3GB ram -> 40 workers ) that simply go out to queued urls and does some light scraping.

They start out great but quickly stall making no more progress. I don't see any errors my logs, just the last item pulled from the queue, the processes remain running, just no more progress being made.

Any suggestions on ways to debug this?

[jrallison]

Hey Jordan,

Humm, that's odd, I haven't seen anything like that. First thing may be to try to figure out where the workers are stalling.

Are the jobs not completing (no "done" log entry if you're using the logging middleware)?
Or are they completing, but no new jobs are picked up off of the redis queue?

The second would definitely be something I'd want to look into. If you can extract the smallest amount of code that reproduces the issue, please post it here so I can help dig into the problem.

Cheers,
John

[vladimir-vg]

I got similar behavior with ruby workers when they just go out of memory. Probably same here.

[jrallison]

Closing for lack of actionable info. Feel free to reopen if that changes.