systemd hides the fact that there exists a maximum unit file line length

[MissingRoberto]

Hello,

I think that the ExecStart in the unit files has a limitation with the length of the command. It cuts the command and indicates "Missing '='.

Does this limit exists really ?

This is the error:

Oct 21 22:01:44 ip-172-31-20-195.ec2.internal systemd[1]: [/run/fleet/units/e-88747910-a986-4ed0-9b10-3af9.minion.service:9] Missing '='.
Oct 21 22:01:45 ip-172-31-20-195.ec2.internal systemd[1]: [/run/fleet/units/e-88747910-a986-4ed0-9b10-3af9.minion.service:8] String is not UTF-8 clean, ignoring assignment: /bin/sh -c "while true; do ; /usr/bin/sleep 3; public_ipv4=$COREOS_PUBLIC_IPV4;  e88747910a9864ed09b103af9_8181=$(docker port e-88747910-a986-4ed0-9b10-3af9 8181 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_8101=$(docker port e-88747910-a986-4ed0-9b10-3af9 8101 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_4444=$(docker port e-88747910-a986-4ed0-9b10-3af9 4444 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_5555=$(docker port e-88747910-a986-4ed0-9b10-3af9 5555 | cut -d ':' -f 2 ); etcdctl set /_esb/user/john/instance/e-88747910-a986-4ed0-9b10-3af9 \"{\\\"ID\\\":\\\"e-88747910-a986-4ed0-9b10-3af9\\\",\\\"Labels\\\":{\\\"owner\\\":\\\"john\\\"},\\\"Created\\\":\\\"Tuesday, 21-Oct-14 21:53:59 UTC\\\",\\\"State\\\":\\\"running\\\",\\\"IP\\\":\\\"$public_ipv4\\\",\\\"Configuration\\\":{\\\"CPU_SHARED\\\":\\\"\\\",\\\"ESB\\\":\\\"servicemix\\\",\\\"Features\\\":[\\\"camel-sql\\\"],\\\"ID\\\":\\\"i-124691ed-f613-4ea5-ab6b-05c7\\\",\\\"JBI_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"JBI_corePoolSize\\\":\\\"4\\\",\\\"JBI_keepAliveTime\\\":\\\"60000\\\",\\\"JBI_maximumPoolSize\\\":\\\"-1\\\",\\\"JBI_queueSize\\\":\\\"1024\\\",\\\"JBI_shutdownTimeout\\\":\\\"0\\\",\\\"JVM_MAX_MEM\\\":\\\"\\\",\\\"JVM_MAX_PERM_MEM\\\":\\\"\\\",\\\"JVM_MIN_MEM\\\":\\\"\\\",\\\"JVM_PERM_MEM\\\":\\\"\\\",\\\"MAX_MEM\\\":\\\"\\\",\\\"NMR_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"NMR_corePoolSize\\\":\\\"4\\\",\\\"NMR_keepAliveTime\\\":\\\"60000\\\",\\\"NMR_maximumPoolSize\\\":\\\"-1\\\",\\\"NMR_queueSize\\\":\\\"1024\\\",\\\"NMR_shutdownTimeout\\\":\\\"0\\\",\\\"Name\\\":\\\"\\\",\\\"Ports\\\":{\\\"endpoint1\\\":4444,\\\"endpoint2\\\":5555},\\\"SERVICEMIX_PASSWORD\\\":\\\"smx\\\",\\\"SERVICEMIX_USER\\\":\\\"smx\\\"},\\\"PortBindings\\\":{\\\"4444\\\":\\\"$e88747910a9864ed09b103af9_4444\\\",\\\"5555\\\":\\\"$e88747910a9864ed09b103af9_5555\\\",\\\"8101\\\":\\\"$e88747910a9864ed09b103af9_8101\\\",\\\"8181\\\":\\\"$e88747910a9864ed09b103af9_8181\\\"},\\\"UserAccess\\\":{\\\"Karaf Console\\\":\\\"$public_ipv4$e88747910a9864ed09b103af

And this is the unit file:

[Unit]
Description=Info docker e-88747910-a986-4ed0-9b10-3af9 ESB Instance
After=e-88747910-a986-4ed0-9b10-3af9.service
Requires=e-88747910-a986-4ed0-9b10-3af9.service

[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c "while true; do ; /usr/bin/sleep 3; public_ipv4=$COREOS_PUBLIC_IPV4;  e88747910a9864ed09b103af9_8181=$(docker port e-88747910-a986-4ed0-9b10-3af9 8181 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_8101=$(docker port e-88747910-a986-4ed0-9b10-3af9 8101 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_4444=$(docker port e-88747910-a986-4ed0-9b10-3af9 4444 | cut -d ':' -f 2 ); e88747910a9864ed09b103af9_5555=$(docker port e-88747910-a986-4ed0-9b10-3af9 5555 | cut -d ':' -f 2 ); etcdctl set /_esb/user/john/instance/e-88747910-a986-4ed0-9b10-3af9 \"{\\\"ID\\\":\\\"e-88747910-a986-4ed0-9b10-3af9\\\",\\\"Labels\\\":{\\\"owner\\\":\\\"john\\\"},\\\"Created\\\":\\\"Tuesday, 21-Oct-14 21:53:59 UTC\\\",\\\"State\\\":\\\"running\\\",\\\"IP\\\":\\\"$public_ipv4\\\",\\\"Configuration\\\":{\\\"CPU_SHARED\\\":\\\"\\\",\\\"ESB\\\":\\\"servicemix\\\",\\\"Features\\\":[\\\"camel-sql\\\"],\\\"ID\\\":\\\"i-124691ed-f613-4ea5-ab6b-05c7\\\",\\\"JBI_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"JBI_corePoolSize\\\":\\\"4\\\",\\\"JBI_keepAliveTime\\\":\\\"60000\\\",\\\"JBI_maximumPoolSize\\\":\\\"-1\\\",\\\"JBI_queueSize\\\":\\\"1024\\\",\\\"JBI_shutdownTimeout\\\":\\\"0\\\",\\\"JVM_MAX_MEM\\\":\\\"\\\",\\\"JVM_MAX_PERM_MEM\\\":\\\"\\\",\\\"JVM_MIN_MEM\\\":\\\"\\\",\\\"JVM_PERM_MEM\\\":\\\"\\\",\\\"MAX_MEM\\\":\\\"\\\",\\\"NMR_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"NMR_corePoolSize\\\":\\\"4\\\",\\\"NMR_keepAliveTime\\\":\\\"60000\\\",\\\"NMR_maximumPoolSize\\\":\\\"-1\\\",\\\"NMR_queueSize\\\":\\\"1024\\\",\\\"NMR_shutdownTimeout\\\":\\\"0\\\",\\\"Name\\\":\\\"\\\",\\\"Ports\\\":{\\\"endpoint1\\\":4444,\\\"endpoint2\\\":5555},\\\"SERVICEMIX_PASSWORD\\\":\\\"smx\\\",\\\"SERVICEMIX_USER\\\":\\\"smx\\\"},\\\"PortBindings\\\":{\\\"4444\\\":\\\"$e88747910a9864ed09b103af9_4444\\\",\\\"5555\\\":\\\"$e88747910a9864ed09b103af9_5555\\\",\\\"8101\\\":\\\"$e88747910a9864ed09b103af9_8101\\\",\\\"8181\\\":\\\"$e88747910a9864ed09b103af9_8181\\\"},\\\"UserAccess\\\":{\\\"Karaf Console\\\":\\\"$public_ipv4$e88747910a9864ed09b103af9_8101\\\",\\\"Web Console\\\":\\\"http://$public_ipv4:$e88747910a9864ed09b103af9_8181/system/console\\\"},\\\"Error\\\":\\\"\\\"}\" --ttl 90; /usr/bin/sleep 30; done"
ExecStop=/usr/bin/echo stopped

[X-Fleet]
MachineOf=e-88747910-a986-4ed0-9b10-3af9.service

[bcwaldon]

@totemteleko The error seems to be "String is not UTF-8 clean, ignoring assignment:", which is at the beginning of the second log line you provided. This can also be reproduced directly with systemd, so this isn't actually a fleet bug. Regardless, I'll definitely help you figure this out.

[MissingRoberto]

Yes, it says about the UTF-8, but it's because it cuts the content.

The line should finish with --ttl 90; /usr/bin/sleep 30; done"
but instead it finishes with :{\"Karaf Console\":\"$public_ipv4$e88747910a9864ed09b103af

The problem appears just with the length, because this file is auto generated and loops in a array of ports, and if I put 2 it works, but not if I add more. In fact, I have the same problem with the other commands: ExecStop, ExecStopPost, etc, and splitting them into several ExecStopPost I don´t get any error.

For example this works because is a shorter modification of the first:

[Unit]
Description=Info docker e-4eb8777c-41bf-4064-9762-679d ESB Instance
After=e-4eb8777c-41bf-4064-9762-679d.service
Requires=e-4eb8777c-41bf-4064-9762-679d.service

[Service]
EnvironmentFile=/etc/environment
ExecStart=/bin/sh -c "while true; do /usr/bin/sleep 3;  etcdctl set /_esb/user/john/instance/e-4eb8777c-41bf-4064-9762-679d \"{\\\"ID\\\":\\\"e-4eb8777c-41bf-4064-9762-679d\\\",\\\"Labels\\\":{\\\"owner\\\":\\\"john\\\"},\\\"Created\\\":\\\"Tuesday, 21-Oct-14 22:31:16 UTC\\\",\\\"State\\\":\\\"running\\\",\\\"IP\\\":\\\"$COREOS_PUBLIC_IPV4\\\",\\\"Configuration\\\":{\\\"CPU_SHARED\\\":\\\"\\\",\\\"ESB\\\":\\\"servicemix\\\",\\\"Features\\\":[\\\"camel-sql\\\"],\\\"ID\\\":\\\"i-124691ed-f613-4ea5-ab6b-05c7\\\",\\\"JBI_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"JBI_corePoolSize\\\":\\\"4\\\",\\\"JBI_keepAliveTime\\\":\\\"60000\\\",\\\"JBI_maximumPoolSize\\\":\\\"-1\\\",\\\"JBI_queueSize\\\":\\\"1024\\\",\\\"JBI_shutdownTimeout\\\":\\\"0\\\",\\\"JVM_MAX_MEM\\\":\\\"\\\",\\\"JVM_MAX_PERM_MEM\\\":\\\"\\\",\\\"JVM_MIN_MEM\\\":\\\"\\\",\\\"JVM_PERM_MEM\\\":\\\"\\\",\\\"MAX_MEM\\\":\\\"\\\",\\\"NMR_allowCoreThreadTimeOut\\\":\\\"true\\\",\\\"NMR_corePoolSize\\\":\\\"4\\\",\\\"NMR_keepAliveTime\\\":\\\"60000\\\",\\\"NMR_maximumPoolSize\\\":\\\"-1\\\",\\\"NMR_queueSize\\\":\\\"1024\\\",\\\"NMR_shutdownTimeout\\\":\\\"0\\\",\\\"Name\\\":\\\"\\\",\\\"Ports\\\":{\\\"endpoint1\\\":4444,\\\"endpoint2\\\":5555},\\\"SERVICEMIX_PASSWORD\\\":\\\"smx\\\",\\\"SERVICEMIX_USER\\\":\\\"smx\\\"},\\\"PortBindings\\\":{\\\"4444\\\":\\\"$(docker port e-4eb8777c-41bf-4064-9762-679d 4444 | cut -d ':' -f 2 )\\\",\\\"5555\\\":\\\"$(docker port e-4eb8777c-41bf-4064-9762-679d 5555 | cut -d ':' -f 2 )\\\",\\\"8101\\\":\\\"$(docker port e-4eb8777c-41bf-4064-9762-679d 8101 | cut -d ':' -f 2 )\\\",\\\"8181\\\":\\\"$(docker port e-4eb8777c-41bf-4064-9762-679d 8181 | cut -d ':' -f 2 )\\\"},\\\"UserAccess\\\":{\\\"Karaf Console\\\":\\\"$COREOS_PUBLIC_IPV4:$(docker port e-4eb8777c-41bf-4064-9762-679d 8101 | cut -d ':' -f 2 )\\\",\\\"Web Console\\\":\\\"http://$COREOS_PUBLIC_IPV4:$(docker port e-4eb8777c-41bf-4064-9762-679d 8181 | cut -d ':' -f 2 )/system/console\\\"},\\\"Error\\\":\\\"\\\"}\" --ttl 90; /usr/bin/sleep 30; done"

ExecStop=/usr/bin/echo stopped

[X-Fleet]
MachineOf=e-4eb8777c-41bf-4064-9762-679d.service

[bcwaldon]

@totemteleko Yes, I believe you're correct. I'm digging into systemd code now to find the limit.

btw it would help readability of your comments if you format code blocks with backticks. I updated your initial bug report as an example.

[bcwaldon]

@totemteleko in my testing, it looks like 2048 characters is the effective maximum. Still trying to grok the code.

[bcwaldon]

Ok, figured it out. The LINE_MAX constant is defined by glibc as 2048. systemd uses LINE_MAX whenever a constant length string is needed, which is ends up using in the config_parse routine. This should definitely be filed as a bug upstream.

[bcwaldon]

Filed here: https://bugs.freedesktop.org/show_bug.cgi?id=85308. @totemteleko Anything else we need to dig into here?

[MissingRoberto]

No, I just wanted to assure that was the problem. Thank you very much.

2014-10-22 2:01 GMT+02:00 Brian Waldon notifications@github.com:

Filed here: https://bugs.freedesktop.org/show_bug.cgi?id=85308.
@totemteleko https://github.com/totemteleko Anything else we need to
dig into here?

—
Reply to this email directly or view it on GitHub
#992 (comment).

Best regards,

Roberto Jiménez Sánchez.

[pirelenito]

I too, just stumbled on this problem.

[bcwaldon]

Lennart confirmed my findings on the upstream systemd bug, so hopefully we make some progress there. I believe the best thing we can do here is to add a line-length check in coreos/go-systemd#69.

[hvolpers]

@totemteleko What if you split the line using backslashes?

[oguzbilgic]

+1 This affected us pretty badly. We added 2 env vars to unit file, and damn we are down.

@hvolpers For some reason, I only hit the limit if I use backslashes with 45 lines.

[lynchc]

+1
http://www.vulcanproxy.com/proxy.html#managing-certificates. The encrypted data is longer than 2048 chars. Once I finally found the issue, I put this in a shell script and executed the script from the unit file

[zeisss]

Systemd allows users to break the 2048 line limit by injecting backslashes and linebreaks. Sadly fleet (I think the coreos/go-systemd/unit) code rejoins them into one line before passing them on to systemd. Just create an ExecStart=echo 0123456789 with 3000 more characters and add a \ and linebreak in the middle. It can't be loaded with fleet.

Interestingly I have to add even more linebreaks pairs to make it working with systemd too (It seems the line length was really ~255 chars with Systemd 208).

=> Can we at least fix this in fleet?

Example file: https://gist.github.com/ZeissS/f02dd7788910aa422e90 (test worked with systemd) - just remove the \ linebreaks.

[pedro-teixeira]

+1, this is a real critical issue

[01101101]

+1

[hectorj2f]

+1

[xh3b4sd]

+1

[jaysonsantos]

I didn't get why fleet strips all line endings, with them systemd can work properly.

[wuqixuan]

+1, fleet should not automatically removing backslashes and linebreaks to hit the systemd bugs.

[zeisss]

@wuqixuan There was a merge last week which should fix that. I am not sure how fast this will end up in fleet though - if you can build fleet yourself you can probably already test it. See coreos/go-systemd#108

[jonboulle]

Right, this needs the dependency to be bumped in fleet.

On Mon, Sep 28, 2015 at 3:58 PM, Stephan notifications@github.com wrote:

@wuqixuan https://github.com/wuqixuan There was a merge last week which
should fix that. I am not sure how fast this will end up in fleet though -
if you can build fleet yourself you can probably already test it. See
coreos/go-systemd#108 coreos/go-systemd#108

—
Reply to this email directly or view it on GitHub
#992 (comment).

[sukrit007]

Just ran into this issue again and breaking lines with "" does not seem to be working. When is the fix planned to be released for this ?

[jonboulle]

There's no current timeline. The fix is at #1375 but I did not merge it
because we might want to consider a strategy for backwards compatibility
(what if people already have such units loaded into their fleet cluster?).
Any suggestions welcome.

On Mon, Nov 16, 2015 at 1:23 PM, Sukrit Khera notifications@github.com
wrote:

Just ran into this issue again and breaking lines with "" does not seem
to be working. When is the fix planned to be released for this ?

—
Reply to this email directly or view it on GitHub
#992 (comment).

[Alino]

+1

[zeisss]

@jonboulle What file do you think may cause breakage? Until now malformed files are not loaded properly by systemd, because the submit step in fleetctl already malformed them afaict. So merging this shouldn't magically fix those old files and lead to execution, as they are still being broken in fleet right? The only change could be that when users destroy and resubmit those files they suddenly work, but I think thats part of fixing bugs ;)

[tixxdz]

The fix at #1375 passes the tests! so I suggest for this one before merging it, create a test that should not fail after #1375 is merged. Then consider the question of backwards compatibility.

[zeisss]

thx for merging!