Explicitly set FD limit

[ikavgo]

Currently we don't set any FD limit, which means we are at mercy of host default settings which means Rabbit can OOM.

Please see here for the context: rabbitmq/cluster-operator#959

What has to be done: Something like CMD ulimit -n 1024 has to be added.

Happy to PR myself. Let me know

[michaelklishin]

1024 is unusable for a distributed data service that often has to sustain a lot of connections. 50K would be my recommended default. Our Production Checklist recommends basing this value on real world metrics but images such as this one cannot follow that route, of course.

[ikavgo]

Thank you @michaelklishin!

[ikavgo]

Also this kubernetes/kubernetes#3595

…

________________________________ From: Michael Klishin ***@***.***> Sent: Thursday, February 17, 2022 7:00:21 PM To: docker-library/rabbitmq ***@***.***> Cc: Iliia Khaprov ***@***.***>; Author ***@***.***> Subject: Re: [docker-library/rabbitmq] Explicitly set FD limit (Issue #545) 1024 is unusable for a distributed data service that often has to sustain a lot of connections. 50K would be my recommended default. Our Production Checklist<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Frabbitmq.com%2Fproduction-checklist.html%23resource-limits-file-handle-limit&data=04%7C01%7Cikhaprov%40vmware.com%7C2b30cd2dd9904ba9bbde08d9f23f5e1a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807176250943653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sordi6%2F3dSe7m3Nx%2Bc%2B9uHMROsiko0OU0WYddYSpADA%3D&reserved=0> recommends basing this value on real world metrics but images such as this one cannot follow that route, of course. — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdocker-library%2Frabbitmq%2Fissues%2F545%23issuecomment-1043248552&data=04%7C01%7Cikhaprov%40vmware.com%7C2b30cd2dd9904ba9bbde08d9f23f5e1a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807176250943653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BhqvPmVltZWyylk%2FC4vQ2C2pQlEBB7k1VZFx9l5Bzdw%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVTKTXOCD5MDFRURBO6BBUDU3UZTLANCNFSM5OVKQK6A&data=04%7C01%7Cikhaprov%40vmware.com%7C2b30cd2dd9904ba9bbde08d9f23f5e1a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807176250943653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8RA3HUafDTW4ZLkDxoevs%2BWeOYiFbsxpprS3kCQvbz8%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cikhaprov%40vmware.com%7C2b30cd2dd9904ba9bbde08d9f23f5e1a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807176250943653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FP3G3CfUpchNgHC%2BzIHi%2BW7RFfVFgD53PfAMP0hD1Ms%3D&reserved=0> or Android<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cikhaprov%40vmware.com%7C2b30cd2dd9904ba9bbde08d9f23f5e1a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807176250943653%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FWzczbofy%2FdR1e1Ff9Kbp2n12ccwknVhCmeNsYI1Bfo%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[tianon]

I think Docker defaults to infinity on the daemon (https://github.com/moby/moby/blob/32e5fe5099d2806df489feb3e15d8e3e6eca73c1/contrib/init/systemd/docker.service#L31), and the limit I get inside a container on my local system isn't quite infinity, but it's still a very high 1048576.

Unfortunately, if we're set to something lower than we want, there's not really much we can do inside the container (besides including a warning in the container logs):

$ docker run -it --rm --ulimit nofile=1024:1024 bash
bash-5.1# ulimit -n 1048576
bash: ulimit: open files: cannot modify limit: Operation not permitted

[ikavgo]

Hi, for me ulimit worked inside kubernetes. I will try inside docker later today. EDIT: I see your are not using rabbitmq image. The ulimit call work for rabbitmq

…

________________________________ From: Tianon Gravi ***@***.***> Sent: Thursday, February 17, 2022 11:52:15 PM To: docker-library/rabbitmq ***@***.***> Cc: Iliia Khaprov ***@***.***>; Author ***@***.***> Subject: Re: [docker-library/rabbitmq] Explicitly set FD limit (Issue #545) I think Docker defaults to infinity on the daemon (https://github.com/moby/moby/blob/32e5fe5099d2806df489feb3e15d8e3e6eca73c1/contrib/init/systemd/docker.service#L31<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmoby%2Fmoby%2Fblob%2F32e5fe5099d2806df489feb3e15d8e3e6eca73c1%2Fcontrib%2Finit%2Fsystemd%2Fdocker.service%23L31&data=04%7C01%7Cikhaprov%40vmware.com%7C68522ce8df704952345608d9f2682568%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807351397326143%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ikgfY6GJQCuoXOmgmXvq82bhlOVUc4inOrcP9tbaTb8%3D&reserved=0>), and the limit I get inside a container on my local system isn't quite infinity, but it's still a very high 1048576. Unfortunately, if we're set to something lower than we want, there's not really much we can do inside the container (besides including a warning in the container logs): $ docker run -it --rm --ulimit nofile=1024:1024 bash bash-5.1# ulimit -n 1048576 bash: ulimit: open files: cannot modify limit: Operation not permitted — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdocker-library%2Frabbitmq%2Fissues%2F545%23issuecomment-1043572208&data=04%7C01%7Cikhaprov%40vmware.com%7C68522ce8df704952345608d9f2682568%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807351397482393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=c%2BakY4FmBb0mLJf00N6909x%2BDDV%2BeQUdXGR0bpw%2Fyj0%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVTKTXIYWTNVXC4CHFVSBATU3V3Z7ANCNFSM5OVKQK6A&data=04%7C01%7Cikhaprov%40vmware.com%7C68522ce8df704952345608d9f2682568%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807351397482393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=7%2BaFWfs%2BjksQrkJs2en1jYDN6pQGRICpwzus7dwAXcg%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Cikhaprov%40vmware.com%7C68522ce8df704952345608d9f2682568%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807351397482393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=wQIlo413oCvwr2%2B9UbtN3oYCyi1zL6cfffVEed%2BTYfU%3D&reserved=0> or Android<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Cikhaprov%40vmware.com%7C68522ce8df704952345608d9f2682568%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637807351397482393%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=gw7N8hIVm4PIbjsSbrhLqZcsksche7cN4ovpCCHabts%3D&reserved=0>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ikavgo]

overriding ERL_MAX_PORTS env var also works. Should we still aim for sane defaults in the Dockerfiles?

[tianon]

I see your are not using rabbitmq image. The ulimit call work for rabbitmq

I'm not sure why it would be any different? This is part of the sandboxing that Docker provides: 😕

$ docker pull rabbitmq
Using default tag: latest
latest: Pulling from library/rabbitmq
Digest: sha256:3d4c70ec5fc84c27efaeb56c50aafcac4fd8583b61398cc028e4876f84ae73d8
Status: Image is up to date for rabbitmq:latest
docker.io/library/rabbitmq:latest

$ docker run -it --rm --ulimit nofile=1024:1024 rabbitmq bash
root@f41cafc78bb2:/# ulimit -n 1048576
bash: ulimit: open files: cannot modify limit: Operation not permitted

Should we still aim for sane defaults in the Dockerfiles?

Yes, absolutely, although I'm still not sure I understand what useful changes we could actually make here (since we can't control the ulimit settings from within the image). 😞

[ikavgo]

[dead@arch eventing-rabbitmq]$ docker run -it rabbitmq bash
root@99778c756269:/# ulimit -Hn
1048576
root@99778c756269:/# ulimit -n 1024
root@99778c756269:/# ulimit -Hn
1024
root@99778c756269:/# 

:-)

(since we can't control the ulimit settings from within the image).

we still can control what portion erlang consumes via ERL_MAX_PORTS

[tianon]

Yes, you can go down, but you cannot go up. 😅 (And the issue here is an environment where the default is too low, right?)

I'm not so sure about adding more shell code to set ERL_MAX_PORTS automatically (nor would I feel comfortable with my own calculations for what to set that to without some kind of "official" guidance 😅).
It seems like a lot for what's essentially a band-aid for a misconfigured host environment. 🙈

[ikavgo]

(And the issue here is an environment where the default is too low, right?)

nope, the issue here was an environment with fd max set to 11billion something

[michaelklishin]

@tianon modern Linux default for file handle limit was last good in 1998, as you know. We should use something at least in the 8K range. Our recommendation to most users is 50K+.

Setting ERL_MAX_PORTS sounds like an acceptable amount of work, especially given how many "non-environment" things env variables were used for in the past. This is an example of a true environment setting we'd offer a sane default for.

[tianon]

As far as I can find looking through RabbitMQ and Erlang source code to try and understand what's the issue here (and how these things are interacting), the relevant bit of RabbitMQ source code is probably https://github.com/rabbitmq/rabbitmq-server/blob/7abf749a60458aba9d6c9e6bdec1d3b0b2254007/deps/rabbit_common/src/file_handle_cache.erl#L1543-L1564 ?

That max_fds value being queried there is ulimit -n, even if ERL_MAX_PORTS is set (on Linux).

On Linux, ERL_MAX_PORTS defaults to 65,536 (in Erlang itself - on Windows, the default limit is a bit lower, but rabbitmq-env.bat sets it to 65,536 if it isn't set explicitly).

So, if ulimit -n is too high (as reported in this issue), I don't see where/how setting ERL_MAX_PORTS would fix it, and I feel like I have to be missing something important. Have we verified that setting ERL_MAX_PORTS explicitly does indeed fix the reported issue?

As far as a fix, I think the best we could do here is something in the entrypoint script (or maybe even in the rabbitmq-server or rabbitmq-env.sh scripts upstream?) that detects if ulimit -n is some fairly arbitrary "too high" value and tries to set it lower? We can't set a default or even a recommended value in the container image -- the best we can do is try to lower it at runtime if it's too high and anything else we have to assume will likely fail with the possibility that even trying to lower it fails (thus ulimit -n xxx || : to make sure we ignore failure). The alternative is to document that setting something like --ulimit nofile=65536 is recommended, but that doesn't work for Kubernetes, as noted above (kubernetes/kubernetes#3595).

(Another trick is that different versions of ulimit disagree on some of the flag names, but I think -n is reasonably consistent, at least between coreutils and BusyBox.)

[lukebakken]

Have we verified that setting ERL_MAX_PORTS explicitly does indeed fix the reported issue?

Yes, see this discussion:

rabbitmq/erlang-rpm#104 (comment)

On Linux, ERL_MAX_PORTS defaults to 65,536 (in Erlang itself - on Windows, the default limit is a bit lower, but rabbitmq-env.bat sets it to 65,536 if it isn't set explicitly).

In both environments, the Erlang VM determines the max number of file descriptors available to it and uses that value. I can't exactly determine how at this time but that's the behavior we see. Grep the erlang/otp source for max_fds.

[tianon]

Ahh, Red Hat is the missing piece of the puzzle here (https://access.redhat.com/solutions/1479623 was specifically enlightening, especially the bits about RHEL 9). I spun up an Alma Linux machine and was finally able to reproduce.

On my Debian system, I cannot increase the nofile limit above 1048576 (no matter what I do, I get "operation not permitted"). On my new Alma Linux 9 VM, my containers are defaulting to 1073741816 and I can reproduce the OOM as described. 🎉

I've now also verified that either of --ulimit nofile=65536 or --env ERL_MAX_PORTS=65536 fix the OOM appropriately.

Given that this is exhibiting outside containers too, I wonder if maybe this is something that scripts/rabbitmq-server should handle in RabbitMQ itself? Are there use cases where huge nofile limits not only work but are desirable? (Would it be reasonable to require those users to set ERL_MAX_PORTS explicitly? Would they be already doing so?)

This is roughly the sort of implementation I had in mind, but it doesn't feel like docker-entrypoint.sh is necessarily the most appropriate place for it (or something like it): 😅

if [ -z "${ERL_MAX_PORTS:-}" ] && nofile="$(ulimit -n)" && [ "$nofile" -gt 65536 ]; then
    export ERL_MAX_PORTS=65536
fi

Should this maybe also consider the amount of memory the system has? Other system load? It feels like it's tough to come up with something generic here that isn't going to unexpectedly break large users who have been up until now perfectly happy with RabbitMQ using an enormous amount of RAM (and have configured it to do so on purpose). Perhaps it would be better/safer to test for 1073741816 explicitly and only handle that known-pathological case specially?

(Would love to hear your thoughts, @michaelklishin ❤️)

[michaelklishin]

I'd go with --env ERL_MAX_PORTS=65536 by default. My gut feeling is that overriding any ulimit would be difficult for the user but overriding an env variable should be straightforward.

We have discovered the same thing in rabbitmq/erlang-rpm#104: on CentOS Stream 9, the default file descriptor limit is dramatically higher, which is generally a good thing but in Erlang's case, increases initial memory allocation significantly.

[michaelklishin]

Testing for 1073741816 sounds good to me. A default of 65K or 100K should be fine. Several VMware products have been using comparable values for years and I don't recall it being an issue.

We should put a visible warning into release notes, too.

[lukebakken]

Given that this is exhibiting outside containers too, I wonder if maybe this is something that scripts/rabbitmq-server should handle in RabbitMQ itself?

This is a good point. @michaelklishin we should consider handling this in rabbitmq-server (cc @dumbbell)

[lukebakken]

See the linked PR for a starting point. I believe we can close the issue here.

[tianon]

Nice, sounds good; thanks! 👍

Closing in favor of rabbitmq/rabbitmq-server#5684 🤘