Rabbitmq boot failed when run in docker rootless

[lystormenvoy]

I install my docker environment according to the document Run the Docker daemon as a non-root user (Rootless mode)

Then I run a rabbitmq container but the rabbitmq server cannot boot.

If I run rabbitmq container in docker with root user and use the same command, it can work.

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 16
  Running: 15
  Paused: 0
  Stopped: 1
 Images: 7
 Server Version: 23.0.4
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: none
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version: v1.1.5-0-gf19387a
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
 Kernel Version: 4.19.91-27.1.an8.x86_64
 Operating System: Anolis OS 8.8
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 7.745GiB
 Name: localhost.localdomain
 ID: ba12bcee-bda1-4d53-84c5-de10e16e0231
 Docker Root Dir: /home/avaapp/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

WARNING: Running in rootless-mode without cgroups. To enable cgroups in rootless-mode, you need to boot the system in cgroup v2 mode.
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Images:

docker logs rabbitmq:

BOOT FAILED
17:41:39.365 [error] ===========
===========
17:41:39.365 [error] ERROR: epmd error for host rabbitmq01: address (cannot connect to host/port)
17:41:39.365 [error] 
ERROR: epmd error for host rabbitmq01: address (cannot connect to host/port)

17:41:40.366 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"rabbitmq01",address} in context start_error
17:41:40.366 [error] CRASH REPORT Process <0.152.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"rabbitmq01",address}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 142
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,\"rabbitmq01\",address}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"rabbitmq01",address}}},{rabbit_prelaunch_app,start,[normal,[]]}}})

hosts in container:
192.168.37.223 is the IP of my server.

root@rabbitmq01:/# cat /etc/hosts 
127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.37.223  rabbitmq01
192.168.37.223  rabbitmq02
192.168.37.223  rabbitmq03
root@rabbitmq01:/# hostname
rabbitmq01

docker run command:

docker run -d --name rabbitmq -v /etc/localtime:/etc/localtime:ro --hostname rabbitmq01 \
  -e RABBITMQ_ERLANG_COOKIE='rabbitmqCookie' --restart always \
  --add-host rabbitmq01:192.168.37.223 \
  --add-host rabbitmq02:192.168.37.223 \
  --add-host rabbitmq03:192.168.37.223 \
  --network=host rabbitmq:3.8.5-management

[lukebakken]

Just so you know, RabbitMQ 3.8 is no longer supported. If this issue turns out to be 3.8 or 3.9-specific there will be no fix.

I will try this out when I have time to set up a rootless Docker environment.

[michaelklishin]

{epmd_error,"rabbitmq01",address} is a weird error tuple, the node fails to start net_kernel or connect to its local epmd. A dive into Erlang/OTP's code would be necessary to see when/where this error tuple is returned exactly.

[lukebakken]

After reading how to setup docker in rootless mode, I don't think I'll be doing that because I don't want to screw up my main development environment.

@lystormenvoy what is the output of this command within the same container you're trying to start RabbitMQ:

ERL_HOSTNAME="$(erl -noinput -eval '{ok,H}=inet:gethostname(),io:format("~s~n", [H]), init:stop().')"; echo "$ERL_HOSTNAME"; echo; ping -c1 "$ERL_HOSTNAME"

If you don't have ping available, install it within the container or install another tool such as nslookup, dig or drill to do a DNS resolution of the hostname that the Erlang VM thinks is the local one.

If you're interested, this is the code that throws the error:

https://github.com/erlang/otp/blob/master/lib/kernel/src/erl_epmd.erl#L149-L156
https://github.com/erlang/otp/blob/master/lib/kernel/src/erl_epmd.erl#L555-L561

[lystormenvoy]

After reading how to setup docker in rootless mode, I don't think I'll be doing that because I don't want to screw up my main development environment.

@lystormenvoy what is the output of this command within the same container you're trying to start RabbitMQ:

ERL_HOSTNAME="$(erl -noinput -eval '{ok,H}=inet:gethostname(),io:format("~s~n", [H]), init:stop().')"; echo "$ERL_HOSTNAME"; echo; ping -c1 "$ERL_HOSTNAME"

If you don't have ping available, install it within the container or install another tool such as nslookup, dig or drill to do a DNS resolution of the hostname that the Erlang VM thinks is the local one.

If you're interested, this is the code that throws the error:

https://github.com/erlang/otp/blob/master/lib/kernel/src/erl_epmd.erl#L149-L156 https://github.com/erlang/otp/blob/master/lib/kernel/src/erl_epmd.erl#L555-L561

The print of your commad:

I found that I can't ping in my container.

[lukebakken]

I think it can't run in rootlesss docker

That's probably not the case, you just have to get networking configured in a way that Erlang works. Are you still getting the exact same error as before? This is the error:

{epmd_error,"rabbitmq01",address}

[lystormenvoy]

I think it can't run in rootlesss docker

That's probably not the case, you just have to get networking configured in a way that Erlang works. Are you still getting the exact same error as before? This is the error:

{epmd_error,"rabbitmq01",address}

yes, it still print:

02:28:35.143 [error] ===========
===========
02:28:35.143 [error] ERROR: epmd error for host rabbitmq01: address (cannot connect to host/port)
ERROR: epmd error for host rabbitmq01: address (cannot connect to host/port)
02:28:35.143 [error] 

02:28:36.145 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason {epmd_error,"rabbitmq01",address} in context start_error
02:28:36.145 [error] CRASH REPORT Process <0.152.0> with 0 neighbours exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"rabbitmq01",address}}},{rabbit_prelaunch_app,start,[normal,[]]}} in application_master:init/4 line 142
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,\"rabbitmq01\",address}}},{rabbit_prelaunch_app,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{epmd_error,"rabbitmq01",address}}},{rabbit_prelaunch_app,start,[normal,[]]}}})

[lukebakken]

OK, when I have the time to set up rootless Docker I will investigate. Obviously the workaround is to use "normal" Docker at this point in time.

[lystormenvoy]

OK, when I have the time to set up rootless Docker I will investigate. Obviously the workaround is to use "normal" Docker at this point in time.

I think I have found that how to resolve, after expose the ports to host then it can work.