Is there a rationale for rabbitmq-server being PID 1 in the pods ? Makes it ignore SIGKILL

[mgarstecki]

Hi,

During an incident where we wanted to force a restart of a RabbitMQ cluster managed by the cluster-operator, we had a hard time forcing a restart of the pods. SIGKILL was getting ignored, and it turns out that's because rabbitmq-server is started as PID 1 in the container, which is allowed to ignore SIGKILL. We resorted to killing the subprocesses of the pod, which pushed rabbitmq-server to crash.

Is there a delibarate rationale to having rabbitmq-server as PID 1, and be able to ignore being killed ? Otherwise, could it be a good idea to wrap it in a simple init process, like tini ?

[lukebakken]

I'm assuming you're using this docker image.

I started it in one shell via:

docker run --pull always --rm --name rabbitmq --publish 5672:5672 --publish 15672:15672 rabbitmq:4-management-alpine

... then in another shell I ran docker kill rabbitmq, and the container exited immediately.

[mgarstecki]

It doesn't exit when the rabbitmq node is under a heavy load. I'm under the impression that it's waiting for other nodes to take leadership on its queues before shutting down, which may take a long time and may never converge.

This is a sane behavior from rabbitmq, to secure data before shutdown. But an operator should be able to kill it, accepting the loss of data, even under this heavy load.

It's also perfectly normal to ignore SIGKILL in a process, since it's normally impossible to handle it... Except when you are PID 1 and actually need to handle it for proper behavior.

[lukebakken]

RabbitMQ has no code in place to handle SIGKILL. If SIGTERM is sent then, yes, a graceful shutdown is attempted, and you would see log messages saying as such.

As of now, the simplest explanation is that the load in your environment caused this issue, not anything that could be done differently in RabbitMQ, its startup scripts, or docker build.

I suggest moving this discussion here - https://github.com/docker-library/rabbitmq/discussions

However, the RabbitMQ docker container has behaved this way since day 1, so I very much doubt it will change.

[Zerpet]

I'm personally a bit against approaches like tiny, because, in my opinion, it's an attempt to convert a container into a VM, which sort of defeats the purpose of a container. RabbitMQ PID does not exactly run as PID 1 in the system, runs as PID 1 inside the container thanks to PID namespaces. Quoting from that documentation link:

The first process created in a new namespace [...] has the PID 1, and is the "init" process for the namespace [...].

Only signals for which the "init" process has established a signal handler can be sent to the "init" process by other members of the PID namespace. [...]

I guess you exec'ed into the container with a bash process, and attempted to kill -9 the process. As stated in the PID NS documentation and shared by Luke, RabbitMQ does not register for SIGKILL, and as such, the signal was ignored.

IMHO, I believe you took the wrong approach. When you want to forcibly kill a container, you don't "enter" the container and start firing shots; instead, you would use the tooling "from the outside" (e.g. docker, nerdctl) to stop the container. Similarly as Luke did docker kill rabbitmq. I understand that this is cumbersome in Kubernetes, because you have to figure out on what node the container is running, connect to such node and then operate. That's assuming that a forceful kubectl delete did not work for some reason.

[mgarstecki]

I agree that it's better to kill the pod from the outside, but having RabbitMQ as PID1 actually breaks that: we tried killing the pods with kubectl delete pod --grace-period 1 (which sends a SIGKILL after one second) and it remained stuck.

kubectl delete pod --grace-period 0 --force caused further issues, the pod having problems restarting until the kubelet cleaned up the leftovers of the previous pod: --grace-period 0 deletes the pod resource from the API server immediately without waiting for subsystems to finish their cleanup, and the new pod starts before the old one is fully deleted on the node.

And some of our clusters are fully-managed by our cloud provider so we can't SSH into the Kubernetes nodes to act straight on the containers either.

rabbitmq-server being PID 1 is clearly causing those limitations. Even if it's a "fake" PID1, being in a container instead of the real init of a VM, it has responsibilities that are unique to being PID1, such as reacting explicitly to SIGKILL. But as a regular program and not an init process, it is reasonable for rabbitmq-server NOT to setup a SIGKILL handler, and it should rely on a robust init process instead.

[lukebakken]

rabbitmq-server being PID 1 is clearly causing those limitations

Considering this is the first report ever of this behavior that I know of, I don't accept that conclusion without a clear reproducer.

My steps showed that the RabbitMQ container does indeed forcefully exit when docker kill is run. Could this be a k8s issue? I have no idea.

If you can provide a way to reproduce this behavior outside of k8s, I suggest starting a discussion here: https://github.com/docker-library/rabbitmq/discussions

Note that it would be sufficient to demonstrate that the rabbitmq-server script (the container entrypoint) receives SIGKILL and ignores it somehow.

Otherwise, the simplest explanation is that this is either specific to your environment, or specific to kubernetes.

[Zerpet]

but having RabbitMQ as PID1 actually breaks that: we tried killing the pods with kubectl delete pod --grace-period 1 (which sends a SIGKILL after one second) and it remained stuck.

It's unclear to me that PID 1 is the root cause. The main idea of managing the container "from the outside" is that rabbitmq process will have a not-1 PID e.g. 12345. You can observe this in a Linux machine by starting a rabbit container with docker run. You will observe a rabbitmq process with a "normal" PID. If you enter the container (via unshare() or via docker exec), then you you will observe PID 1. If you send signals from within the container, all your theory makes sense; that's why it's critically important to manage the containers from the outside, as you correctly did with kubectl delete pod.

As Luke mentioned, I believe the root cause of your issue is somewhere else, not in rabbit having PID 1.

Think of this from a different perspective: if your claim were true, any and all containers that don't wrap their process with tiny or similar, would be broken and "unkillable" because their processes (e.g. mysqld, httpd, nginx, postgres) do not handle their "init responsibilities". A quick experiment will demonstrate that this is not the case, and that all of those containers can be forcibly terminated by either of the kubectl commands you shared.

[lukebakken]

Hi @mgarstecki -

I've had this issue in the back of my head ever since you reported it. I'm not a k8s expert by any means, so I had a discussion with "the genie" (i.e. ChatGPT) about this issue. Here is what it had to say.

I do agree with the genie that the fact that RabbitMQ running as PID 1 is not the issue, because, external to the container, everything runs in a cgroup that the kernel can easily can kill. I did find it interesting what the AI had to say about the load on the system and how, perhaps, it could have lead to the appearance that SIGKILL is being ignored when it's something else in the k8s internals. You certainly know more about that so I hope you find it interesting.

Finally, there is a pretty easy way to run the Erlang VM (beam.smp) as PID 1, rather than the rabbitmq-server script. If you set RUNNING_UNDER_SYSTEMD=true or detached=true in the environment, the script will exec the Erlang VM so that it will become PID 1 (link).

I tested this out locally via this command:

docker run --pull always --rm --env=RUNNING_UNDER_SYSTEMD=true --name rabbitmq --publish 5672:5672 --publish 15672:15672 rabbitmq:latest

Since the Erlang VM traps SIGTERM and gracefully stops (including stopping RabbitMQ applications), running docker stop rabbitmq works as expected.

The reason I'm suggesting this is that, maybe, running RabbitMQ as PID 1 could avoid any issues that could arise by the rabbitmq-server script remaining as PID 1.

I think there's a case to be made for the docker container's behavior to change so that beam.smp is PID 1. I'll bring it up there.

[lukebakken]

docker-library/rabbitmq#778

@Zerpet input requested on the above PR, if you'd like. Thanks.