99% CPU on ubuntu 12.04 when clients connect #144

Closed
CodeMonkeyKevin opened this Issue Oct 7, 2012 · 14 comments

Projects

None yet

4 participants

@CodeMonkeyKevin

Beanstalkd process is showing 99% cpu on ubuntu 12.04 (fresh install) when 2 or more clients connect.

I'm using beanstalkd 1.7.

@kr
Owner
kr commented Oct 8, 2012

I tried to reproduce this using the description you gave,
but was unsuccessful. Can you provide more detailed
steps to reproduce the problem?

@CodeMonkeyKevin

I upgraded from 1.4.6 to 1.7 and got this issue.

I think this maybe an issue with the ruby beanstalkd-client gem. I'll do some more testing and report back.

@kr
Owner
kr commented Oct 8, 2012

If you run version 1.7 with -VVV you'll get some detailed
logs. Could you post a sample of the logs you see during
the 99% cpu usage? There's no need to capture huge
amounts of logs; just a few hundred lines should suffice.

@CodeMonkeyKevin

Awesome, I ran it with the -VVV and got the 99% CPU usage again right away.

Full log: http://pastie.org/4931070

@catwell

If that can help, I have the same issue with 1.7 on CentOS (actually Amazon Linux), and I do not use the Ruby gem on the client side.

Logs:

pid 2862
bind 6 127.0.0.1:11300
accept 11
<11 command use 
>11 reply USING $dandelion$-uploads
<11 command watch 
>11 reply WATCHING 2
<11 command ignore 
>11 reply WATCHING 1
<11 command reserve

The 100% CPU usage actually occurs not when I connect the consumer client but when I quit it.

@CodeMonkeyKevin

I downgraded to 1.4.6 and got the same issue. So the issue must be with a dependency.

@CodeMonkeyKevin

KR where you able to replicate the issue? Using EC2 official ubuntu/centos AMIs i get this issue right away.

@kr
Owner
kr commented Oct 10, 2012

I've reproduced it with 1.7 on Ubuntu 12.04. However, the bug
does not manifest in head of master. I'll track down which
commit fixed it; regardless, the next release will not have this
bug.

@kr kr closed this Oct 10, 2012
@kr
Owner
kr commented Oct 10, 2012

Oops, that's wrong. It definitely happens on master.

@kr kr reopened this Oct 10, 2012
@kr
Owner
kr commented Oct 10, 2012

For reference, a further reduced test case is:

  1. Start beanstalkd
  2. Connect a client
  3. Run reserve
  4. Disconnect the client
@CodeMonkeyKevin

This issue is a little strange. I have one server (lets call it server-a) with ubuntu 10.04 that doesn't have this issue, all the other other linux distro and even the clones of the good server server-a i tested gets this issue.

@CodeMonkeyKevin

kr, any luck tracking down this issue? I been trying to fix it by self with no luck. Seems to be effecting all versions after 1.4.6.

@jaydiablo

We're seeing this as well on a CentOS 5 box running in EC2 with version 1.7. We never saw 1.4.6 eat this much CPU (running on the same size EC2 instance, but not the same instance itself). Queue activity hasn't changed from 1.4.6 to 1.7.

@kr
Owner
kr commented Oct 18, 2012

This bug was introduced in a9d4367.

The problem is that when the client sends reserve, we currently
ask for an event when the client hangs up, but don't actually close
the connection. As long as the connection remains open, this
hangup event is always available. Not only does this waste CPU,
it also effectively leaks an fd.

I have a fix prepared, once I get a chance to test the fix more,
I'll push it out. Probably tomorrow.

@kr kr added a commit that closed this issue Oct 19, 2012
@kr correctly check for hangup during reserve
Revert a9d4367.

Instead of closing the connection immediately when the client
hangs up, merely cancel any pending reserve command. We still
process any other commands that are available so that a client
can read responses if the connection is only half closed.

fixes #144
c2ee0f3
@kr kr closed this in c2ee0f3 Oct 19, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment