99% CPU on ubuntu 12.04 when clients connect

[CodeMonkeyKevin]

Beanstalkd process is showing 99% cpu on ubuntu 12.04 (fresh install) when 2 or more clients connect.

I'm using beanstalkd 1.7.

[kr]

I tried to reproduce this using the description you gave,
but was unsuccessful. Can you provide more detailed
steps to reproduce the problem?

[CodeMonkeyKevin]

I upgraded from 1.4.6 to 1.7 and got this issue.

I think this maybe an issue with the ruby beanstalkd-client gem. I'll do some more testing and report back.

[kr]

If you run version 1.7 with -VVV you'll get some detailed
logs. Could you post a sample of the logs you see during
the 99% cpu usage? There's no need to capture huge
amounts of logs; just a few hundred lines should suffice.

[CodeMonkeyKevin]

Awesome, I ran it with the -VVV and got the 99% CPU usage again right away.

Full log: http://pastie.org/4931070

[catwell]

If that can help, I have the same issue with 1.7 on CentOS (actually Amazon Linux), and I do not use the Ruby gem on the client side.

Logs:

pid 2862
bind 6 127.0.0.1:11300
accept 11
<11 command use 
>11 reply USING $dandelion$-uploads
<11 command watch 
>11 reply WATCHING 2
<11 command ignore 
>11 reply WATCHING 1
<11 command reserve

The 100% CPU usage actually occurs not when I connect the consumer client but when I quit it.

[CodeMonkeyKevin]

I downgraded to 1.4.6 and got the same issue. So the issue must be with a dependency.

[CodeMonkeyKevin]

KR where you able to replicate the issue? Using EC2 official ubuntu/centos AMIs i get this issue right away.

[kr]

I've reproduced it with 1.7 on Ubuntu 12.04. However, the bug
does not manifest in head of master. I'll track down which
commit fixed it; regardless, the next release will not have this
bug.

[kr]

Oops, that's wrong. It definitely happens on master.

[kr]

For reference, a further reduced test case is:

1. Start beanstalkd
2. Connect a client
3. Run reserve
4. Disconnect the client

[CodeMonkeyKevin]

This issue is a little strange. I have one server (lets call it server-a) with ubuntu 10.04 that doesn't have this issue, all the other other linux distro and even the clones of the good server server-a i tested gets this issue.

[CodeMonkeyKevin]

kr, any luck tracking down this issue? I been trying to fix it by self with no luck. Seems to be effecting all versions after 1.4.6.

[jaydiablo]

We're seeing this as well on a CentOS 5 box running in EC2 with version 1.7. We never saw 1.4.6 eat this much CPU (running on the same size EC2 instance, but not the same instance itself). Queue activity hasn't changed from 1.4.6 to 1.7.

[kr]

This bug was introduced in a9d4367.

The problem is that when the client sends reserve, we currently
ask for an event when the client hangs up, but don't actually close
the connection. As long as the connection remains open, this
hangup event is always available. Not only does this waste CPU,
it also effectively leaks an fd.

I have a fix prepared, once I get a chance to test the fix more,
I'll push it out. Probably tomorrow.