Hanging jobs?

[bmulholland]

I tried this out locally and have seen that some jobs appear to get stuck in Busy. It's not consistent, so I can't 100% say that this isn't happening without staged-push, but I've reproduced the issue 2-3 times with staged-push and never without it.

Is there anything that staged-push does that could cause this? Anything I can do to debug?

[bmulholland]

We use ActiveJob, if that helps

[adamniedzielski]

Thank you for reporting 💛 I'm not using this gem in production anywhere so any reports of the usage are useful for me.

What do you mean by:

some jobs appear to get stuck in Busy

? Are they being picked up by Sidekiq from Redis for execution and then never complete?

Is there anything that staged-push does that could cause this? Anything I can do to debug?

Nothing comes to my mind. The only thing that could block Sidekiq is this extra thread - https://github.com/adamniedzielski/sidekiq-staged_push/blob/bb186973fba42991850fa610c6bd21de038f1e7e/lib/sidekiq/staged_push/enqueuer.rb. We call sleep so it shouldn't be preventing other threads from running 🤔

[bmulholland]

I'm not using this gem in production

This makes me wary, TBH.

Are they being picked up by Sidekiq from Redis for execution and then never complete?

Yep, looks that way.

[adamniedzielski]

This makes me wary, TBH.

Very good approach, I approve 👍 Somebody needs to start using it, though, so we can discover the issues and fix them. The theoretical foundations for this gem are solid - https://brandur.org/job-drain. If something doesn't work then it has to be my silly implementation mistakes and they should be easy to fix.

I wish I could try it out in production on my current team. We use https://github.com/que-rb/que and there aren't many pragmatic reasons to switch to Sidekiq + sidekiq-staged_push. As an individual contributor I could try pushing it through just for the sake of my own curiosity, but I manage this team so I can't 😛.

https://github.com/adamniedzielski/sidekiq-staged_push-demo is the demo app and sandbox for the gem. @bmulholland Could you open a PR there where you add the job where you can reproduce the issue somewhat reliably? Then it's easy for me to take a look.

[adamniedzielski]

Closing because of inactivity. Folks can share their experience in #14