Prioritize systemd-remount-fs for local-fs.target #5358

davispuh · 2017-02-15T20:01:20Z

Submission type

Bug report
Request for enhancement (RFE)

systemd version the issue has been seen with

systemd 232

Used distribution

Arch Linux

In case of bug report: Expected behaviour you didn't see

systemd-remount-fs didn't start because unrelated automount was masked for local-fs.target
Initramfs mounted root fs read-only so / was unwritable causing pretty much all services to fail (due /var). It was supposed to be remounted read-write by systemd-remount-fs
I think systemd-remount-fs should be run in any case even if other automounts for local-fs.target fail.

$ systemctl status systemd-remount-fs.service
● systemd-remount-fs.service - Remount Root and Kernel File Systems
   Loaded: loaded (/usr/lib/systemd/system/systemd-remount-fs.service; static; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:systemd-remount-fs.service(8)

$ systemctl status local-fs.target
● local-fs.target - Local File Systems
   Loaded: loaded (/usr/lib/systemd/system/local-fs.target; static; vendor preset: disabled)
   Active: active since Wed 2017-02-15 21:37:40 EET; 3min 35s ago
     Docs: man:systemd.special(7)

feb 15 21:37:40 host systemd[1]: Reached target Local File Systems.
feb 15 21:37:52 host systemd[1]: local-fs.target: Cannot add dependency job, ignoring: Unit mnt-Data.automount is masked.
feb 15 21:37:52 host systemd[1]: local-fs.target: Cannot add dependency job, ignoring: Unit mnt-Data.automount is masked.

$ systemctl list-dependencies local-fs.target
local-fs.target
● ├─-.mount
● ├─mnt-Data.automount
● ├─systemd-remount-fs.service
● ├─tmp.mount
● ├─tmp.mount
● ├─usr.mount
● └─var-lib-machines.mount

In case of bug report: Steps to reproduce the problem

Have root / fs mounted read-only by initramfs and have /etc/fstab

/dev/sda1     /                 btrfs  rw,autodefrag,compress=lzo                    0 0
/dev/sdb1     /mnt/Data         btrfs  rw,compress=zlib,noauto,x-systemd.automount   0 0

$ systemctl mask mnt-Data.automount
Created symlink /etc/systemd/system/mnt-Data.automount → /dev/null.

Idea is that you can quickly mask/unmask without needing to edit fstab.

After reboot you can't even unmask it without manually remounting...

$ systemctl unmask mnt-Data.automount
Failed to unmask unit: File /etc/systemd/system/mnt-Data.automount: Read-only file system

The text was updated successfully, but these errors were encountered:

poettering · 2017-02-16T10:27:34Z

Please provide a longer boot log excerpt.

systemd-remote-fs.service does not actually depend on any local file systems, it is only ordered after systemd-fsck-root.service, that's all. You can verify that easily by looking at the "systemctl show systemd-remote-fs.service" output.

davispuh · 2017-02-16T20:08:03Z

emmm I'm talking about systemd-remount-fs.service not systemd-remote-fs.service
What kind of log you want to see?

Compare without masked mnt-Data.automount

$ systemd-analyze critical-chain --fuzz=1h local-fs.target
local-fs.target @3.780s
├─tmp.mount @3.776s +3ms
│ ├─local-fs-pre.target @3.775s
│ │ ├─systemd-tmpfiles-setup-dev.service @2.858s +916ms
│ │ │ ├─kmod-static-nodes.service @1.588s +248ms
│ │ │ │ ├─system.slice
│ │ │ │ │ └─-.slice
│ │ │ │ └─systemd-journald.socket
│ │ │ │   ├─-.slice
│ │ │ │   │ └─...
│ │ │ │   └─-.mount
│ │ │ │     └─system.slice
│ │ │ │       └─...
│ │ │ ├─system.slice
│ │ │ │ └─...
│ │ │ └─systemd-journald.socket
│ │ │   └─...
│ │ └─systemd-remount-fs.service @1.624s +1.176s
│ │   ├─systemd-fsck-root.service @584542y 2w 2d 20h 1min 41.278s +19ms
│ │   │ ├─system.slice
│ │   │ │ └─...
│ │   │ └─systemd-journald.socket
│ │   │   └─...
│ │   ├─system.slice
│ │   │ └─...
│ │   └─systemd-journald.socket
│ │     └─...
│ ├─system.slice
│ │ └─...
│ └─-.mount
│   └─...
├─local-fs-pre.target @3.775s
│ └─...
├─systemd-remount-fs.service @1.624s +1.176s
│ └─...
├─mnt-Data.automount @1.585s
│ └─-.mount
│   └─...
├─systemd-fsck-root.service @584542y 2w 2d 20h 1min 41.278s +19ms
│ └─...
└─-.mount
  └─...

With masked mnt-Data.automount (systemd-remount-fs.service doesn't start)
Looks like local-fs-pre.target doesn't start either.

$ systemd-analyze critical-chain --fuzz=1h local-fs.target
local-fs.target @1.790s
├─tmp.mount @1.594s +190ms
│ ├─system.slice
│ │ └─-.slice
│ └─-.mount
│   └─system.slice
│     └─...
├─systemd-fsck-root.service @584542y 2w 2d 20h 1min 40.846s +22ms
│ ├─systemd-journald.socket
│ │ ├─-.slice
│ │ │ └─...
│ │ └─-.mount
│ │   └─...
│ └─system.slice
│   └─...
└─-.mount
  └─...

poettering · 2017-02-17T09:26:06Z

emmm I'm talking about systemd-remount-fs.service not systemd-remote-fs.service

Sorry, I mistyped, that's what I meant.

poettering · 2017-02-17T09:27:31Z

please provide the boot log output. "journalctl -b", ideally when booted with "systemd.log_level=debug" on the kernel cmdline.

davispuh · 2017-02-17T19:02:08Z

$ systemctl show systemd-remount-fs.service
[...]
Requires=system.slice
Wants=local-fs-pre.target
WantedBy=local-fs.target
Conflicts=shutdown.target
Before=local-fs.target systemd-sysusers.service local-fs-pre.target shutdown.target systemd-coredump@0.service systemd-timesyncd.service systemd-update-utmp.service systemd-rfkill.service systemd-journal-flush.service systemd-hwdb-update.service systemd-random-seed.service systemd-firstboot.service
After=systemd-fsck-root.service systemd-journald.socket system.slice
[...]

Full log here https://paste.kde.org/piogbxuag

feb 17 19:42:57 pchost systemd[1]: local-fs.target: Cannot add dependency job, ignoring: Unit mnt-Data.automount is masked.

dpisklov · 2017-08-13T13:00:39Z

Have same experience, in my case tmp.mount was masked and that (silently!) stopped systemd-remount-fs.service from being executed. Only after coming across this issue, I noticed the
systemd[1]: local-fs.target: Cannot add dependency job, ignoring: Unit tmp.mount is masked.
warning in logs. So I unmasked it and voila - remount worked after reboot.
Logs and stuff are all very similar to what @davispuh has provided.

poettering · 2017-08-31T14:39:32Z

systemd[1]: local-fs.target: Cannot add dependency job, ignoring: Unit tmp.mount is masked.

Well, as this says, the fact that this unit is masked is actually ignored...

poettering · 2017-08-31T14:41:26Z

Full log here https://paste.kde.org/piogbxuag

hmm, this is strange, remount-fs is not even mentioned in here at all

poettering · 2017-08-31T14:42:07Z

something really strange is happening here...

makhomed · 2023-04-03T21:43:39Z

Have same experience, in my case tmp.mount was masked and that (silently!) stopped systemd-remount-fs.service from being executed.

I have the same problem on the fresh Rocky Linux 9.1 installation with systemd 250 (250-12.el9_1.3). But I can't find any warnings or error messages in the logs, I just see the systemd-remount-fs.service in the dead state, and root filesystem leave mounted in the read only mode, without any comments. Only your github message help me to resolve this problem, Thank you!

When I unmasked tmp.mount - remount worked via systemd-remount-fs.service after reboot.

Recommendation to mask tmp.mount I found at the freedesktop.org site:

https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems/
Disable any mounting on /tmp so that it resides on the same physical file system as the root directory. For that, execute systemctl mask tmp.mount

dtardon · 2023-04-04T06:18:45Z

Have same experience, in my case tmp.mount was masked and that (silently!) stopped systemd-remount-fs.service from being executed.

I have the same problem on the fresh Rocky Linux 9.1 installation with systemd 250 (250-12.el9_1.3). But I can't find any warnings or error messages in the logs, I just see the systemd-remount-fs.service in the dead state, and root filesystem leave mounted in the read only mode, without any comments. Only your github message help me to resolve this problem, Thank you!

When I unmasked tmp.mount - remount worked via systemd-remount-fs.service after reboot.

Recommendation to mask tmp.mount I found at the freedesktop.org site:

We've never had a separate /tmp on RHEL by default, so what you're doing isn't even necessary. (Unless Rocky reverts the change.)

makhomed · 2023-04-04T12:31:00Z

We've never had a separate /tmp on RHEL by default, so what you're doing isn't even necessary. (Unless Rocky reverts the change.)

It was my mistake - in the kickstart file I completely disable swap, create separate /tmp partition, and disable tmp.mount to prevent (as I previously think) creation of tmpfs which can use up to 50% of available RAM.

And result was completely unexpected: when I disable tmp.mount in the %post section of kickstart file - after installation of Enterprise Linux 9.1 operating system - root filesystem was mounted in read only mode, sshd daemon not started and server was not usable and not accessible from network.

From my user point of view disabling tmp.mount is completely unrelated to mounting root filesystem in read only mode.

But from systemd 250 point of view - if tmp.mount is disabled - the systemd-remount-fs.service must not be started and root filesystem must be leaved in the read only mode. Unexpected. Very unexpected. May be this is bug in the systemd or RHEL9.1 ?

Now I fix my own mistake - I create 256 GiB swap for server with 128 GiB RAM and enable tmp.mount in the %post section of the kickstart file for creating separate filesystem for /tmp using tmpfs using up to 50% of installed RAM.

P.S.

Red Hat Enterprise Linux 9.1 is the best Linux distro in the entire world.

Rocky Linux 9.1 is 100% compatible with the Red Hat Enterprise Linux 9.1, but without the need to spend time and effort on games with licensing of the installed operating system - without support licensing games is useless and only take time and effort.

Rocky Linux 9.1 did not reverts any changes in the distro, it is 100% compatible with RHEL 9.1

poettering · 2023-04-04T17:00:08Z

OK, as I understand this was a local issue, closing then.

dtardon · 2023-04-05T08:08:51Z

OK, as I understand this was a local issue, closing then.

No, I just noted to @makhomed that he'd run into the issue because he'd done something pointless (in context of RHEL's default configuration). But the issue is completely generic. I just reproduced it on F-34 with latest main:

mask boot.mount
reboot
/ is read-only because systemd-remount-fs.service has not run

dtardon · 2023-04-05T08:10:06Z

Output of journalctl -b

poettering added the needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer label Feb 16, 2017

nghiant2710 mentioned this issue Nov 8, 2017

Fix #323, fix issue of broken systemd (/dev/console) in docker container balena-io-library/base-images#342

Merged

keszybz removed the needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer label Nov 8, 2018

poettering added the units label Nov 20, 2018

poettering closed this as completed Apr 4, 2023

dtardon reopened this Apr 5, 2023

dtardon added the bug 🐛 Programming errors, that need preferential fixing label Apr 5, 2023

YHNdnzj added the remount-fs label Apr 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prioritize systemd-remount-fs for local-fs.target #5358

Prioritize systemd-remount-fs for local-fs.target #5358

davispuh commented Feb 15, 2017 •

edited

poettering commented Feb 16, 2017

davispuh commented Feb 16, 2017

poettering commented Feb 17, 2017 •

edited

poettering commented Feb 17, 2017

davispuh commented Feb 17, 2017

dpisklov commented Aug 13, 2017

poettering commented Aug 31, 2017

poettering commented Aug 31, 2017

poettering commented Aug 31, 2017

makhomed commented Apr 3, 2023

dtardon commented Apr 4, 2023

makhomed commented Apr 4, 2023

poettering commented Apr 4, 2023

dtardon commented Apr 5, 2023

dtardon commented Apr 5, 2023

Prioritize systemd-remount-fs for local-fs.target #5358

Prioritize systemd-remount-fs for local-fs.target #5358

Comments

davispuh commented Feb 15, 2017 • edited

Submission type

systemd version the issue has been seen with

Used distribution

In case of bug report: Expected behaviour you didn't see

In case of bug report: Steps to reproduce the problem

poettering commented Feb 16, 2017

davispuh commented Feb 16, 2017

poettering commented Feb 17, 2017 • edited

poettering commented Feb 17, 2017

davispuh commented Feb 17, 2017

dpisklov commented Aug 13, 2017

poettering commented Aug 31, 2017

poettering commented Aug 31, 2017

poettering commented Aug 31, 2017

makhomed commented Apr 3, 2023

dtardon commented Apr 4, 2023

makhomed commented Apr 4, 2023

poettering commented Apr 4, 2023

dtardon commented Apr 5, 2023

dtardon commented Apr 5, 2023

davispuh commented Feb 15, 2017 •

edited

poettering commented Feb 17, 2017 •

edited