systemd-resolved won't start on read-only filesystem, unless run outside systemd

[Malvineous]

Submission type

• Bug report
• Request for enhancement (RFE)

systemd version the issue has been seen with

systemd version 232, Arch Linux ARM (systemd 232-8) on Raspberry Pi

Expected behaviour you didn't see

/run/systemd/resolve/resolve.conf was not created, even though /run is on tmpfs and writable.

Unexpected behaviour you saw

systemd-resolved fails to launch:

# systemctl restart systemd-resolved
Job for systemd-resolved.service failed because of unavailable resources or another system error.
See "systemctl status systemd-resolved.service" and "journalctl -xe" for details.

# systemctl status systemd-resolved
* systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
   Active: failed (Result: resources) since Wed 2017-02-01 11:13:07 AEST; 4s ago
     Docs: man:systemd-resolved.service(8)
           http://www.freedesktop.org/wiki/Software/systemd/resolved
           http://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           http://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

Feb 01 11:13:07 rpi systemd[1]: systemd-resolved.service: Unit entered failed state.

Ignore the last line (dated Feb 01) - since the filesystem is read only, no new log entries can be created. This means journalctl -xe provides no further details either.

However systemd-resolved appears to work fine when run directly:

# /usr/lib/systemd/systemd-resolved 
Positive Trust Anchors:
. IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Using system hostname 'rpi'.

# ls /run/systemd/resolve/resolv.conf 
-rw-r--r-- 1 systemd-resolve systemd-resolve 547 Feb  1 11:12 /run/systemd/resolve/resolv.conf

It only seems to be that when systemd-resolved is started by systemd itself that it fails. None of the other systemd services are failing.

For the record, remounting the root filesystem read/write (with mount / -o remount,rw) so that the journal can be viewed makes systemctl restart systemd-resolved succeed, so there are no errors recorded in the journal anyway.

Steps to reproduce the problem

1. Mark the root filesystem as readonly (in /etc/fstab and the kernel command line)
2. Confirm /run or wherever systemd-resolved needs to store its files is read-write, e.g. by putting a tmpfs mount here. This is the default for Arch Linux ARM.
3. Reboot
4. cat /run/systemd/resolve/resolv.conf will return a file not found error
5. Run /usr/lib/systemd/systemd-resolved and observe the cat command in the previous step now works, even though the root filesystem is still read-only.

[evverx]

The root cause is PrivateTmp=, which requires writable /var/tmp.

1     mkdir("/tmp/systemd-private-5459b4fd21bf455cbe6dff3198caa4b3-systemd-resolved.service-wxnfyZ", 0700) = 0
1     umask(000)                        = 077
1     umask(000)                        = 000
1     mkdir("/tmp/systemd-private-5459b4fd21bf455cbe6dff3198caa4b3-systemd-resolved.service-wxnfyZ/tmp", 01777) = 0
1     umask(000)                        = 000
1     umask(077)                        = 000
1     mkdir("/var/tmp/systemd-private-5459b4fd21bf455cbe6dff3198caa4b3-systemd-resolved.service-JQU9yc", 0700) = -1 EROFS (Read-only file system)
1     umask(000)                        = 077
1     rmdir("/tmp/systemd-private-5459b4fd21bf455cbe6dff3198caa4b3-systemd-resolved.service-wxnfyZ/tmp") = 0
1     rmdir("/tmp/systemd-private-5459b4fd21bf455cbe6dff3198caa4b3-systemd-resolved.service-wxnfyZ") = 0
1     sendmsg(37<UNIX:[9534]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="PRIORITY=4\nSYSLOG_FACILITY=3\nCODE_FILE=src/core/service.c\nCODE_LINE=1845\nCODE_FUNC=service_enter_start\nERRNO=30\nUNIT=systemd-resolved.service\n7ae3f2e300f04aee94cb42458041aa28INVOCATION_ID=\nSYSLOG_IDENTIFIER=systemd\n", iov_len=215}, {iov_base="MESSAGE=", iov_len=8}, {iov_base="systemd-resolved.service: Failed to run 'start' task: Read-only file system", iov_len=75}, {iov_base="\n", iov_len=1}], msg_iovlen=4, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 299

[evverx]

The regression was introduced by 0c28d51.

[Malvineous]

Thanks for this - mounting tmpfs in /var/tmp fixed the problem for me. Not sure whether this is actually a bug in systemd or whether it's a distro/config fault that /var/tmp is not writable.

[poettering]

I am pretty sure we should expect that /var/tmp is writable during normal operation. We expect the same from /run and /tmp, and /var as a whole. Closing.

[nagualcode]

There is no reason a software should fail because of this. Give a warning, and resume.

[Arjun765]

I am pretty sure we should expect that /var/tmp is writable during normal operation. We expect the same from /run and /tmp, and /var as a whole. Closing.

But should time syncing fail because the filesystem is read only?