[3.6.13] - Service file contains inline comments after systemd parameters

[mctwynne]

With the release of 3.6.13 it looks like comments were added to the systemd service file units, which is not permitted. From what I understand they must be on new lines only.

If the Arch wiki is anything to go by: "Tip: Comments prepended with # may be used in unit-files as well, but only in new lines. Do not use end-line comments after systemd parameters or the unit will fail to activate."

After installing 3.6.13 I get the following errors in syslog:

[/lib/systemd/system/rabbitmq-server.service:17] Failed to parse service restart specifier, ignoring: on-failure # https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=

[/lib/systemd/system/rabbitmq-server.service:18] Failed to parse sec value, ignoring: 10 # https://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartSec=

[/lib/systemd/system/rabbitmq-server.service:23] Failed to parse value, ignoring: # rabbitmq/rabbitmq-server-release#51

[/lib/systemd/system/rabbitmq-server.service:23] Failed to parse value, ignoring: rabbitmq/rabbitmq-server-release#51

During this process the service restarts in a loop from what I can tell.

[michaelklishin]

It would help immensely if you provided at least some information about your distribution, systemd version and such.

Somehow the packages pass a test suite on a dozen of platforms. Anyhow, it is easy enough to remove those comments.

[mctwynne]

@michaelklishin I'm running Ubuntu 16.04, and I installed rabbitmq via puppet+packagecloud as part of an OpenStack deployment. Is there any more info you need from me? What do you mean by systems version?

After reading more about comments in service files it seems inline comments are not permitted by systemd: "Empty lines and lines starting with "#" or ";" are ignored. This may be used for commenting." Inline comments are not permitted. Since I'm doing a puppet deployment, removing the comments is easier said then done on my end since I'm deploying to lots of hosts, and the rest of the installation requires that rabbit is functional before continuing.

Thanks for looking into this.

[michaelklishin]

I meant systemd version, a word not known to autocomplete.

I removed all inline comments and I guess will be producing an emergency release now.

[mctwynne]

@michaelklishin Thank you very much!!

[michaelklishin]

Hm, probably not until I QA rabbitmq/rabbitmq-management#508 and the pipeline tells me things are not obviously broken (hey, they are looking OK without this change already).

[mctwynne]

Just FYI, systemd --version gives me: systemd 229

I'm not sure how this would have passed tests and not reloaded the service constantly since it's unsupported usage of service files, at least with my version of systemd.

[michaelklishin]

The tests start, restart and stop the service plus do some basic sanity checking.

[michaelklishin]

@mctwynne is there any chance you can try the version currently in stable?

It will likely take about 2 hours for the pipeline to pass and looks like producing a final (or even one-off package) may have to wait until tomorrow (Nov 8th) because I'm on the road and have limited access to our infrastructure.

[mctwynne]

@michaelklishin I don't know if or how I can specify a specific version with puppet. I haven't found a way yet. Any chance you might know?

[michaelklishin]

@mctwynne I didn't find anything that would override the .service file in the module. I am going to upload a one-off package here, at least you should be able to try it with dpkg -i.

[mctwynne]

If you just want to know if removing the inline comments works, it does. I've deleted them and restarted the service, and no longer get the errors.

[mctwynne]

@michaelklishin I'm assuming that would be the only change in the package you'd upload?

[michaelklishin]

That and rabbitmq/rabbitmq-management#508.

[michaelklishin]

@mctwynne please give this a try

rabbitmq-server-3.6.13.901-deb.zip

[michaelklishin]

I don't have 16.04 handy but our package tests that cover some 7-8 distributions that use systemd still pass and I did some manual testing on 17.04. Works like a charm.

I also tested 3.6.13 manually and somehow I cannot reproduce this issue on 17.04. We will discuss doing an emergency 3.6.14 release with the team shortly.

[michaelklishin]

FTR, my test VM has systemd 232.

[dumbbell]

Here is what is reported on our Ubuntu 16.04 VM used for testing packages:

# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-11-07 08:44:20 UTC; 2min 18s ago
 Main PID: 22579 (beam.smp)
   Status: "Initialized"
   CGroup: /system.slice/rabbitmq-server.service
           ├─22579 /usr/lib/erlang/erts-9.1/bin/beam.smp -W w -A 64 -P 1048576 -t 5000000 -stbt db -zdbbl 128000 -K true -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.6.13/ebin -noshell -noinput -s rabbit b
           ├─22659 /usr/lib/erlang/erts-9.1/bin/epmd -daemon
           ├─22793 erl_child_setup 1024
           ├─22815 inet_gethost 4
           └─22816 inet_gethost 4

Nov 07 08:44:18 ip-172-31-30-31 rabbitmq-server[22579]:   ##########
Nov 07 08:44:18 ip-172-31-30-31 rabbitmq-server[22579]:               Starting broker...
Nov 07 08:44:20 ip-172-31-30-31 rabbitmq-server[22579]: systemd unit for activation check: "rabbitmq-server.service"
Nov 07 08:44:20 ip-172-31-30-31 systemd[1]: Started RabbitMQ broker.
Nov 07 08:44:20 ip-172-31-30-31 rabbitmq-server[22579]:  completed with 0 plugins.
Nov 07 08:44:20 ip-172-31-30-31 systemd[1]: [/lib/systemd/system/rabbitmq-server.service:17] Failed to parse service restart specifier, ignoring: on-failure # https://www.freedesktop.org/software/systemd/man/systemd.service.html#Restart=
Nov 07 08:44:20 ip-172-31-30-31 systemd[1]: [/lib/systemd/system/rabbitmq-server.service:18] Failed to parse sec value, ignoring: 10      # https://www.freedesktop.org/software/systemd/man/systemd.service.html#RestartSec=
Nov 07 08:44:20 ip-172-31-30-31 systemd[1]: [/lib/systemd/system/rabbitmq-server.service:23] Failed to parse value, ignoring: # rabbitmq/rabbitmq-server-release#51
Nov 07 08:44:20 ip-172-31-30-31 systemd[1]: [/lib/systemd/system/rabbitmq-server.service:23] Failed to parse value, ignoring: rabbitmq/rabbitmq-server-release#51
Nov 07 08:45:55 ip-172-31-30-31 systemd[1]: Started RabbitMQ broker.

So indeed the errors appear in the log, however they don't prevent the service from starting. That's why the errors remain unnoticed. Perhaps systemd can run in a stricter mode; I'll do some research to understand why it fails to start for you and not us.

[dumbbell]

I found systemd-analyze(1) which offers a verify command to lint Unit files. I'll add it to our testsuite.

[michaelklishin]

3.6.14 is out, thanks to @dumbbell our test suite now includes system spec file validation.