Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

v245-stable: ordering cycle reported, prevents non-root login or failed automount #69

Closed
andyrtr opened this issue Jun 6, 2020 · 9 comments
Closed

v245-stable: ordering cycle reported, prevents non-root login or failed automount #69

andyrtr opened this issue Jun 6, 2020 · 9 comments

Comments

@andyrtr
Copy link

andyrtr commented Jun 6, 2020

Using Arch Linux we run into a new bug in stable release 245.6 that introduces a "Ordering cycle reported" leading to random services not properly loaded and login or automount failing.

Bisected to this commit and reverting it seems to fix it:
e1c091b

Downstream report: https://bugs.archlinux.org/task/66908
@cdown
Copy link
Member

cdown commented Jun 7, 2020 •
edited
Loading

Seems I don't have rights in systemd-stable repo to transfer issues, @poettering, @keszybz, any chance you can hook me up? :-)

Since this is a straight backport of systemd/systemd@b3d7aef, this is something we should ideally centralise discussion of on the main repo.

It's not immediately obvious to me why that change would cause this problem, but probably it's exposing some other overall dependency issue. cc @fbuihuu, who has more context.

@fbuihuu
Copy link
Contributor

fbuihuu commented Jun 8, 2020

From a quick glance, it seems that all problematic cases make the automount unit use x-systemd.requires=network-online.target.

This doesn't look correct since automount units are supposed to be ordered before local-fs.target.

@cdown do you know why this requirement is needed ?

@fbuihuu
Copy link
Contributor

fbuihuu commented Jun 8, 2020

Something strange is that the ordering cycle should have already existed if the automount unit was generated by fstab-generator, which seems to be the case from one report.

@cdown
Copy link
Member

cdown commented Jun 8, 2020

From a quick glance, it seems that all problematic cases make the automount unit use x-systemd.requires=network-online.target.

Right, but shouldn't this then also have happened with fstab-generator before?

This doesn't look correct since automount units are supposed to be ordered before local-fs.target.

@cdown do you know why this requirement is needed ?

I assume it's a well-intentioned attempt to make network shares contingent on the network. Certainly it feels like the way we expose this issue to users (hey, you can't log in!) is probably the worst manifestation I could conjure, though. It would be nice if we could surface this somewhere earlier.

@fbuihuu
Copy link
Contributor

fbuihuu commented Jun 8, 2020

Right, but shouldn't this then also have happened with fstab-generator before?

That's what is puzzling me currently (see my previous comment).

I did give it a try on my archlinux VM and I still could see the ordering cycle with systemd-243.162-2-x86_64.pkg.tar.xz (that was the latest version available locally which doesn't have the suspected commits):

Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found ordering cycle on systemd-resolved.service>
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on systemd-tmpfiles-setup.servi>
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on local-fs.target/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on mnt.automount/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on network-online.target/start
Jun 08 17:53:21 archlinux-vm systemd[1]: network.target: Found dependency on network.target/start

But for some reasons no jobs was deleted in order to break this cycle here, which was the case when I tested systemd-245.6-4-x86_64.pkg.tar.zst...

@fbuihuu
Copy link
Contributor

fbuihuu commented Jun 8, 2020

I assume it's a well-intentioned attempt to make network shares contingent on the network.

But that seems wrong - automount units are supposed to be ordered before local-fs.target so also ordering them after network-online.target will just create ordering cycles and a (random) job to be killed during the boot process.

I think only mount units should depend on the network but ordering against network-online.target is probably the wrong target to choose. And fstab-generator is supposed to add the network deps automatically anyway (you can also force it to do so with _netdev option).

@cdown
Copy link
Member

cdown commented Jun 9, 2020

Sure, I don't think it actually does what they expect, but it's well-intentioned. I can see why people would think they should do that.

cdown added a commit to cdown/systemd that referenced this issue Jun 9, 2020
@cdown
doc: Try to clarify automount dependency confusion
f0eaa3a 
Arch recently upgraded systemd to 245.6. Shortly afterwards, users began
reporting[0] that systemd detected an ordering cycle, and they were
unable to log in. The reason they were unable to log in was because of
ordering cycle resolution:

    [...]
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    [...]

Whether the resolution did the right thing here or not is a longer-term
discussion, but in the interim we should at least make this distinction
between automount dependencies and mount dependencies clearer in the
documentation, so that users and distribution maintainers know what's
acceptable. In this case Arch actually backed out b3d7aef entirely and
released a new version due to the confusion.

Also see systemd/systemd-stable#69.

0: https://bugs.archlinux.org/task/66908
@cdown cdown mentioned this issue Jun 9, 2020
Merged
@fbuihuu
Copy link
Contributor

fbuihuu commented Jun 9, 2020

Hmm... wait.

If an entry in fstab uses the following options: x-systemd.automount,x-systemd.requires=network-online.target.

Is the dependency on network-online.target supposed to be applied to the mount unit or to the automount one ?

Currently it applies to the automount unit but I'm not sure if this is really intended...

@poettering @keszybz ?

poettering pushed a commit to systemd/systemd that referenced this issue Jun 9, 2020
@cdown @poettering
doc: Try to clarify automount dependency confusion
69876f9 
Arch recently upgraded systemd to 245.6. Shortly afterwards, users began
reporting[0] that systemd detected an ordering cycle, and they were
unable to log in. The reason they were unable to log in was because of
ordering cycle resolution:

    [...]
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    [...]

Whether the resolution did the right thing here or not is a longer-term
discussion, but in the interim we should at least make this distinction
between automount dependencies and mount dependencies clearer in the
documentation, so that users and distribution maintainers know what's
acceptable. In this case Arch actually backed out b3d7aef entirely and
released a new version due to the confusion.

Also see systemd/systemd-stable#69.

0: https://bugs.archlinux.org/task/66908
fbuihuu added a commit to fbuihuu/systemd that referenced this issue Jun 11, 2020
@fbuihuu
fstab-generator: extra dependencies specified in fstab should be appl…
0aee9e1 
…ied to the mount unit

If an entry in fstab uses "x-systemd.automount" option and also asks for
additionnal dependencies via x-systemd.requires or such, then the dependencies
were applied to the automount unit.

But this unlikely to do the right thing and is consistent with network mount
network mounts.

Indeed for entries specifying "_netdev,x-systemd.automount" options, both a
network mount is generated as well as an automount one. But the network
dependencies are only applied to the mount unit, the automount unit remains
ordered against local-fs.target.

The same logic should be followed when extra deps are specified via the mount
options as automount units should always be ordered against local-fs.target.

Note: in general such extra deps should be used with care and should specify
dependencies on other mount units only. Indeed it can easily create ordering
cycles like it's been seen in
systemd/systemd-stable#69 as mount units (as well as
automount ones) are ordered before local-fs.target which is a low-level target
that most other units depend on.
@fbuihuu fbuihuu mentioned this issue Jun 11, 2020
Merged
fbuihuu added a commit to fbuihuu/systemd that referenced this issue Jul 1, 2020
@fbuihuu
fstab-generator: extra dependencies specified in fstab should be appl…
045c5fa 
…ied to the mount unit

If an entry in fstab uses "x-systemd.automount" option and also asks for
additionnal dependencies via x-systemd.requires or such, then the dependencies
were applied to the automount unit.

But this unlikely to do the right thing and is inconsistent with what's done
for network mounts.

Indeed when an fstab entries has "_netdev,x-systemd.automount" options, the
dependencies against the network requested by "_netdev" are (correctly) applied
to the mount unit only and the automount unit remains ordered against
local-fs.target.

The same logic should be followed when extra deps are specified via the mount
options as automount units should always be ordered against local-fs.target.

Note: in general explicit deps specified via mount options should be used with
care and should be used to specify dependencies on other mount units only as it
can easily create ordering cycles otherwise like it's been seen in
systemd/systemd-stable#69. Mount units (as well as
automount ones) are ordered before local-fs.target by default which is a
low-level target that most other units depend on.
keszybz pushed a commit that referenced this issue Jul 27, 2020
@cdown @keszybz
doc: Try to clarify automount dependency confusion
a2d46b6 
Arch recently upgraded systemd to 245.6. Shortly afterwards, users began
reporting[0] that systemd detected an ordering cycle, and they were
unable to log in. The reason they were unable to log in was because of
ordering cycle resolution:

    [...]
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    [...]

Whether the resolution did the right thing here or not is a longer-term
discussion, but in the interim we should at least make this distinction
between automount dependencies and mount dependencies clearer in the
documentation, so that users and distribution maintainers know what's
acceptable. In this case Arch actually backed out b3d7aef entirely and
released a new version due to the confusion.

Also see #69.

0: https://bugs.archlinux.org/task/66908
(cherry picked from commit 69876f9)
keszybz pushed a commit that referenced this issue Jul 27, 2020
@fbuihuu @keszybz
fstab-generator: extra dependencies specified in fstab should be appl…
bc740e3 
…ied to the mount unit

If an entry in fstab uses "x-systemd.automount" option and also asks for
additionnal dependencies via x-systemd.requires or such, then the dependencies
were applied to the automount unit.

But this unlikely to do the right thing and is inconsistent with what's done
for network mounts.

Indeed when an fstab entries has "_netdev,x-systemd.automount" options, the
dependencies against the network requested by "_netdev" are (correctly) applied
to the mount unit only and the automount unit remains ordered against
local-fs.target.

The same logic should be followed when extra deps are specified via the mount
options as automount units should always be ordered against local-fs.target.

Note: in general explicit deps specified via mount options should be used with
care and should be used to specify dependencies on other mount units only as it
can easily create ordering cycles otherwise like it's been seen in
#69. Mount units (as well as
automount ones) are ordered before local-fs.target by default which is a
low-level target that most other units depend on.

(cherry picked from commit 045c5fa)
sayanchowdhury pushed a commit to kinvolk-archives/systemd-legacy that referenced this issue Aug 3, 2020
@cdown @sayanchowdhury
doc: Try to clarify automount dependency confusion
aa29824 
Arch recently upgraded systemd to 245.6. Shortly afterwards, users began
reporting[0] that systemd detected an ordering cycle, and they were
unable to log in. The reason they were unable to log in was because of
ordering cycle resolution:

    [...]
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-update-done.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-journal-catalog-update.service/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job local-fs.target/start deleted to break ordering cycle starting with sysinit.target/start
    systemd[1]: sysinit.target: Job systemd-tmpfiles-setup.service/start deleted to break ordering cycle starting with sysinit.target/start
    [...]

Whether the resolution did the right thing here or not is a longer-term
discussion, but in the interim we should at least make this distinction
between automount dependencies and mount dependencies clearer in the
documentation, so that users and distribution maintainers know what's
acceptable. In this case Arch actually backed out b3d7aef entirely and
released a new version due to the confusion.

Also see systemd/systemd-stable#69.

0: https://bugs.archlinux.org/task/66908
(cherry picked from commit 69876f9)
sayanchowdhury pushed a commit to kinvolk-archives/systemd-legacy that referenced this issue Aug 3, 2020
@fbuihuu @sayanchowdhury
fstab-generator: extra dependencies specified in fstab should be appl…
90c9859 
…ied to the mount unit

If an entry in fstab uses "x-systemd.automount" option and also asks for
additionnal dependencies via x-systemd.requires or such, then the dependencies
were applied to the automount unit.

But this unlikely to do the right thing and is inconsistent with what's done
for network mounts.

Indeed when an fstab entries has "_netdev,x-systemd.automount" options, the
dependencies against the network requested by "_netdev" are (correctly) applied
to the mount unit only and the automount unit remains ordered against
local-fs.target.

The same logic should be followed when extra deps are specified via the mount
options as automount units should always be ordered against local-fs.target.

Note: in general explicit deps specified via mount options should be used with
care and should be used to specify dependencies on other mount units only as it
can easily create ordering cycles otherwise like it's been seen in
systemd/systemd-stable#69. Mount units (as well as
automount ones) are ordered before local-fs.target by default which is a
low-level target that most other units depend on.

(cherry picked from commit 045c5fa)
@keszybz keszybz changed the title Ordering cycle reported, prevents non-root login or failed automount v245-stable: ordering cycle reported, prevents non-root login or failed automount Dec 8, 2020
@keszybz
Copy link
Member

keszybz commented Dec 10, 2020

v245.7 has bc740e3, let's close this.

@keszybz keszybz closed this as completed Dec 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fbuihuu @keszybz @cdown @andyrtr