Beanstalk does not free memory

[wimrijnders]

Hi,

We're using beanstalk in a high-throughput application, in which tens if not hundreds of jobs per second are processed in multiple queues in beanstalkd.

Inevitably, the beanstalkd process takes over the entire memory of the server it's running on. The rate at which this happens varies, it can take days or, just like now, half an hour.

This looks an awful lot like issue #54, only that's closed now. Since this behaviour is affecting our application, I would like to readdress the issue.

I was wondering, has any headway been made with issue #54? Perhaps you can advise me as to how to avoid the problem. Please tell what what other info I can supply.

Thanks,

Wim.

[tylerhi]

+1

[l3x]

+1

[tchwpkgorg]

+1

[gopi-ar]

+1, @kr we're facing this issue as well on Ubuntu 14.04. Restarting the beanstalkd process is the only way out so far. Any updates?

[jbergstroem]

You guys should probably provide more information if you want this issue to move forward. It's impossible to triage this in its current state. How about dumping memory with gdb and having a quick look at what's allocated compared to when it doesn't grow?

[wimrijnders]

I am acutely embarrassed to discover this issue is still open; I will take some time soon to make a more concrete case for this issue.

[Lujeni]

+1

[btall]

+1

[ultrabug]

+1, same kind of workload, same memory-not-released issue

[wimrijnders]

Hi guys, thanks for putting up with the delay. I finally put some effort into examining the bug.

On writing this comment, I find I have a lot to say. I'll keep it as succinct as possible.

---

Application

Our project consists of one ruby-based webserver, and many distributed components written in C++.
There is one central beanstalkd instance, which handles multiple tubes for all components.

The webserver uses beaneater to interface with beanstalkd. The C++ code uses beanstalk-client.

When the application runs at full tilt, putting 1000 jobs/second on a tube is common. So when memory issues are encountered, these can escalate very quickly.

The beanstalk version used:

> beanstalkd -v
beanstalkd 1.10

Issues

To reiterate the problems that occurred:

1. Memory allocated in beanstalkd appeared to be not released. Given enough workload, the memory usage increased monotonically.
2. On mass-deletion of tube contents, the amount of memory used increased significantly, till deletion had been completed.

Issue 2. is explicitly noted because it has been a pain; given reasonable regular memory usage, on deletion of a large number of jobs (eg. aborting a given distributed task), the amount of memory used exploded till the RAM was maxed out and pandemonium ensued.

Current solution

I solved this issue in our code by throttling the amount of jobs sent to beanstalkd. I.e. the number of jobs put on a tube is balanced as well as possible by the number of jobs detected to be read out. This works well, but is of course a workaround for the actual issue.

Example code

I've tried recreating the issue with a simple test program, with limited success.

The test consists of a script send.rb, which writes many jobs in bursts to a tube. Another script receive.rb reads out the jobs. In the meantime, I keep track of beanstalkd memory usage with top.

I was continually tweaking the code during testing, to see if I could completely re-create the issues mentioned. Most of these tweaks were useless and have been removed.

Observations

Issue 2

For the life of me, I can't reproduce it with the given code. This might have to do with me being lazy and doing a ruby test program. I might have to do something similar in C++.

Issue 1

I can reproduce this partially. The memory usage of beanstalkd increases up to a certain limit, which I think is related to the max number of jobs in the tube at any point.

Pre: restart of beanstalkd, relevant top output:

%CPU  %MEM     TIME+ COMMAND
0.00  0.004  0:00.02 beanstalkd

Post: after running scripts for some time:

%CPU  %MEM    TIME+ COMMAND
30.33 29.97 0:40.62 beanstalkd 

This memory is never freed again, even if the send script is terminated so that the receive script can consume all jobs. And then, when the receive script is also terminated, the memory still is not freed.

However, contrary to initial comment, it does NOT continue increasing.

Conclusions and Follow-up

So I couldn't fully exemplify the issues with ruby code. This might be due to:

• the issues being present in the C++-based client.
• It might also have to with raw processing speeds, assuming that C++ will be a damn sight faster than the corresponding ruby code.

I also note that the example code is overly simplistic; IRL there are many clients writing to and reading from a given tube. I tried simulating this by running an extreme number of copies of the given scripts, without much difference in results.

So I might have to bite the bullet and make a similar test program with C++.

---

OK, so this sort of turned into a scientific essay. I hope it's of use to you.

[wimrijnders]

Ad previous:

Writing that, well, 'essay', prompted me to think further about the problem. Regarding Issue 2, I think I understand now what was happening.

There were multiple tubes being read/written to (think of responses coming in for sent requests); I had a single loop reading out all tubes in sequence - this has evolved into separate reading threads by now.

Anyway, when a task got aborted, the outstanding jobs were deleted; this happened in the loop. So, while deleting, other tubes were being written to at 1000 jobs/second while not being read from. So it's to be expected that memory will be filling up really fast.

This adequately explains the memory explosion issue. So, I hereby retract Issue 2.

[Shir0kamii]

Hi,

If someone is still interested, I've found out what was the problem and fixed it. Turns out the culprit was the hash map. The patch is available on #295 or on numberly's fork.

Best regards

[dbpolito]

I'm looking forward for this too! :(

[silviucpp]

This fix was merged in master for some time.
Silviu

[JensRantil]

@silviucpp Has there been a new release of beanstalkd released with the new fix?

[dbpolito]

I hasn't... last release is 1.10 which doesn't contain this fix.

[silviucpp]

There is no release and as time there is no activity from maintainer and nobody else can do a release we won't have one soon. Project seems dead so only way is to compile yourself the master and maintain your own fork.

[JensRantil]

@kr Could you please make a release with the fix so someone can check if this improves memory usage?

[Shir0kamii]

Hello everyone,

We have a fork of beanstalk at work with the fix I made (http://github.com/numberly/beanstalkd) and we run it in production.

I can tell you that if your beanstalk server's volume of message never go significantly lower than usual, it doesn't improve the memory usage by much.

However, if your beanstalk is sometime empty, the fix can be a great improvement in memory usage as it will reduce the hasmap size to its minimum.

I hope it helps!

[silviucpp]

Fix for this was already merged in master. b7b4a6a

Can be closed

[ysmolski]

Great, I have doubled checked. Thanks for pointing to these issues!

[imashok02]

I see theres no official release for the below config,
its still
beanstalkd 1.10

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.5 LTS"

Did anyone downloaded from the official website beanstalkd and install the version with the fix? I not sure how to do that.

[tomponline]

@imashok02 beanstalkd 1.11 is in Ubuntu Focal (which is the latest LTS release) https://packages.ubuntu.com/focal/beanstalkd

You could post on the Ubuntu forums asking for a later version to be backported, but I am not sure what the policies are for that.