Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sidekiq up, jobs not processing #849

Closed
cmer opened this issue Apr 14, 2013 · 8 comments
Closed

Sidekiq up, jobs not processing #849

cmer opened this issue Apr 14, 2013 · 8 comments

Comments

@cmer
Copy link

cmer commented Apr 14, 2013

It's happened to us 4 times in the last 2 weeks where we had plenty of jobs in our queue (~10k) but Sidekiq is not processing them. ps aux says sidekiq 2.9.0 foobar [0 of 20 busy]. The web interface is just "stale" as well.

This never happened before and seems to have started happening around the same time we upgraded pg to 0.15.0 from 0.14.1. I doubt it's related, though.

I don't have very many details unfortunately.
@cmer
Copy link
Author

cmer commented Apr 14, 2013

This is Sidekiq 2.9.0 but also happened with 2.6.5.

@cmer
Copy link
Author

cmer commented Apr 14, 2013

Oh, and I noticed that sometimes the old Sidekiq process doesn't die when deploying with Capistrano. Not sure if this is all linked. I deployed last night which seems to be right around when the workers stopped processing jobs.

@mperham
Copy link
Collaborator

mperham commented Apr 14, 2013

Does the log file contain any errors or backtraces?

On Sun, Apr 14, 2013 at 7:25 AM, Carl Mercier notifications@github.comwrote:

It's happened to us 4 times in the last 2 weeks where we had plenty of
jobs in our queue (~10k) but Sidekiq is not processing them. ps aux says sidekiq
2.9.0 foobar [0 of 20 busy]. The web interface is just "stale" as well.

This never happened before and seems to have started happening around the
same time we upgraded pg to 0.15.0 from 0.14.1. I doubt it's related,
though.

I don't have very many details unfortunately.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/849
.

@cmer
Copy link
Author

cmer commented Apr 14, 2013

Nothing particular, unfortunately.

@cmer
Copy link
Author

cmer commented Apr 15, 2013

Ok this just happened again.

Nothing in the log file. I took some screenshots. When killing Sidekiq with Capistrano, it says it needed to be shutdown forcefully.

Sidekiq-1
hipyard-web deus web1_ ssh 202 48
hipyard-web bash 126 48-1

@cmer
Copy link
Author

cmer commented Apr 15, 2013

For the record, I did not deploy before this happened, so it's probably not related to deploys at all.

@cmer
Copy link
Author

cmer commented Apr 15, 2013

What's really odd is that the UI says I have a bunch of workers running and ps aux says 2. The web UI says the jobs were started 14 minutes ago, but that's simply not possible given what these tasks are. I feel 14 minutes ago is when Sidekiq stopped processing my jobs instead.

@cmer cmer mentioned this issue Apr 15, 2013
Closed
@mperham
Copy link
Collaborator

mperham commented Apr 19, 2013

Closing as a dupe of #702

@mperham mperham closed this as completed Apr 19, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cmer @mperham