Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CentOS 7 is broken (in dokken) #435

Closed
rmoriz opened this issue Apr 22, 2017 · 6 comments

Comments

Projects
None yet
1 participant
@rmoriz
Copy link
Contributor

commented Apr 22, 2017

(you can skip to the bottom or just read how I waste my weekends…)

service[rabbitmq-server] action start
hangs until chef-client exits.

See:
https://travis-ci.org/rabbitmq/chef-cookbook/jobs/221400408

I can reproduce this issue on dokken/docker.

@rmoriz rmoriz referenced this issue Apr 22, 2017

Closed

Chef 13 support #70

@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2017

[root@01b3d8c9ee66 /]# systemctl status rabbitmq-server.service
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
   Active: activating (start) since Sat 2017-04-22 14:49:26 UTC; 14min ago
 Main PID: 3169 (beam.smp)
   CGroup: /docker/01b3d8c9ee668b5396d2d374bb279181567a5f73840432192d5bd9bb62b14eea/system.slice/rabbitmq-server.service
           ├─3169 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -- -root /usr/li...
           ├─3393 inet_gethost 4
           └─3394 inet_gethost 4
           ‣ 3169 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -- -root /usr/li...

Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: ##  ##
Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: ##########  Logs: /var/log/rabbitmq/rabbit@01b3d8c9ee66.log
Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: ######  ##        /var/log/rabbitmq/rabbit@01b3d8c9ee66-sasl.log
Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: ##########
Apr 22 14:49:28 01b3d8c9ee66 rabbitmq-server[3169]: Starting broker...
Apr 22 14:49:30 01b3d8c9ee66 rabbitmq-server[3169]: systemd unit for activation check: "-.slice"
Apr 22 14:49:30 01b3d8c9ee66 rabbitmq-server[3169]: Unexpected status from systemd "systemctl: invalid option -- '.'\n"
Apr 22 14:49:30 01b3d8c9ee66 rabbitmq-server[3169]: systemd READY notification failed, beware of timeouts
Apr 22 14:49:30 01b3d8c9ee66 rabbitmq-server[3169]: completed with 0 plugins.

-> notify to systemd fails
-> startup never works

@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2017

https://github.com/rabbitmq/rabbitmq-server/blob/rabbitmq_v3_6_8/src/rabbit.erl#L377
executes:

[root@01b3d8c9ee66 /]# systemctl status 3169
● -.slice - Root Slice
   Loaded: loaded (/usr/lib/systemd/system/-.slice; static; vendor preset: disabled)
   Active: active since Sat 2017-04-22 14:34:03 UTC; 33min ago
     Docs: man:systemd.special(7)
   CGroup: /docker/01b3d8c9ee668b5396d2d374bb279181567a5f73840432192d5bd9bb62b14eea
           ├─1 /usr/lib/systemd/systemd
           └─system.slice
             ├─dbus.service
             │ └─63 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
             ├─rabbitmq-server.service
             │ ├─3169 /usr/lib64/erlang/erts-5.10.4/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 32000 -K true -- -root /us...
             │ ├─3393 inet_gethost 4
             │ └─3394 inet_gethost 4
             ├─system-epmd.slice
             │ └─epmd@0.0.0.0.service
             │   └─3281 /usr/bin/epmd -systemd
             ├─system-getty.slice
             │ └─getty@tty1.service
             │   └─81 /sbin/agetty --noclear tty1 linux
             ├─systemd-logind.service
             │ └─59 /usr/lib/systemd/systemd-logind
             ├─systemd-udevd.service
             │ └─26 /usr/lib/systemd/systemd-udevd
             └─systemd-journald.service
               └─17 /usr/lib/systemd/systemd-journald

=> "_.slice"

@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2017

https://github.com/rabbitmq/rabbitmq-server/blob/rabbitmq_v3_6_8/src/rabbit.erl#L400

executes

[root@01b3d8c9ee66 /]# systemctl show --property=ActiveState -.slice
systemctl: invalid option -- '.'
@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2017

Bug in Rabbitmq. They don't do proper shell-escaping…

systemctl show --property=ActiveState \\-.slice
ActiveState=inactive
@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 22, 2017

#436 makes rabbit start and report to systemd using socat/sd_notify.

BUT

tests are still broken due to a 2,5 year old serverspec version that relies on netstat (deprecated with CentOS 7) to retrieve port bindings and also breaks with non-ASCII content (which is the dot in systemctl status ...)

😭

I'll try to convert them to inspec and provide PR tomorrow.

@rmoriz

This comment has been minimized.

Copy link
Contributor Author

commented Apr 23, 2017

Will continue in #434

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.