Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multipathd/socket: start conditions not aligned with service unit, funky result #15

Closed
lucab opened this issue Sep 21, 2021 · 2 comments · Fixed by #20
Closed

multipathd/socket: start conditions not aligned with service unit, funky result #15

lucab opened this issue Sep 21, 2021 · 2 comments · Fixed by #20

Comments

@lucab
Copy link
Contributor

lucab commented Sep 21, 2021

Right now the service unit multipathd.service has several start conditions that the socket unit multipathd.socket does not have. It would be good to align the set of conditions across the two units.

The current diff is:

> ConditionKernelCommandLine=!nompath
> ConditionKernelCommandLine=!multipath=off
> ConditionVirtualization=!container

This means that if one of the conditions is not met (e.g. kargs contains nompath), a client could trigger the socket unit to start, then enter a retry loop, and eventually be marked as failed. The socket unit has not other possible outcome than ending up in a failed state, because it is indeed impossible to start the related service.

As a real example, a random client command like multipathd show daemon will send the multipathd.socket unit into this failed state:

* multipathd.socket - multipathd control socket
     Loaded: loaded (/usr/lib/systemd/system/multipathd.socket; enabled; vendor preset: disabled)
     Active: failed (Result: trigger-limit-hit) since Tue 2021-09-21 14:37:45 UTC; 17s ago
   Triggers: * multipathd.service
     Listen: @/org/kernel/linux/storage/multipathd (Stream)

Sep 21 14:37:45 coreos-scratch systemd[1]: multipathd.socket: Trigger limit hit, refusing further activation.
Sep 21 14:37:45 coreos-scratch systemd[1]: multipathd.socket: Failed with result 'trigger-limit-hit'.

This happens because the unit is trying really hard to activate the service unit and forward the client request to it. However, multipathd.service is instead in the following state, where it will (correctly) never manage to actually start (despite all the retries):

* multipathd.service - Device-Mapper Multipath Device Controller
     Loaded: loaded (/usr/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Tue 2021-09-21 14:33:08 UTC; 4min 54s ago
TriggeredBy: x multipathd.socket
  Condition: start condition failed at Tue 2021-09-21 14:37:45 UTC; 17s ago
             `- ConditionKernelCommandLine=!nompath was not met
   Main PID: 476 (code=exited, status=0/SUCCESS)
        CPU: 21ms

Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
Sep 21 14:37:45 coreos-scratch systemd[1]: Condition check resulted in Device-Mapper Multipath Device Controller being skipped.
@mwilck
Copy link
Contributor

mwilck commented Sep 21, 2021

We can change this in multipath-tools, but it sounds like a systemd issue to me. The same would apply for any socket-activated unit. I can see no documented requirement that the conditions in socket units and their associated service units must be equal.

@lucab
Copy link
Contributor Author

lucab commented Sep 21, 2021

PR at #16.

Yes I think it's a bit of an unfortunate corner-case of systemd dependency graph, but I believe this comes from the deliberate decision of keeping unit settings as self-contained as possible (i.e. not infecting other units).

mwilck pushed a commit to openSUSE/multipath-tools that referenced this issue Sep 28, 2021
This aligns 'multipathd' socket and service units, by adding the
start conditions that are set on the service but not on the socket.
It should help avoiding situations where the socket unit ends up
marked as failed after hitting its retry-limit.

Fixes: opensvc#15
Signed-off-by: Luca BRUNO <luca.bruno@coreos.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
bmarzins pushed a commit to bmarzins/rh-multipath-tools that referenced this issue Oct 7, 2021
Upstream Status: https://github.com/openSUSE/multipath-tools.git
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2008101
Conflicts: Match the conditions with RHEL-8 multipathd.service unit

commit 345ccf5
Author: Luca BRUNO <luca.bruno@coreos.com>
Date:   Fri Sep 24 09:34:01 2021 +0000

    multipathd.socket: add missing conditions from service unit

    This aligns 'multipathd' socket and service units, by adding the
    start conditions that are set on the service but not on the socket.
    It should help avoiding situations where the socket unit ends up
    marked as failed after hitting its retry-limit.

    Fixes: opensvc/multipath-tools#15
    Signed-off-by: Luca BRUNO <luca.bruno@coreos.com>
    Reviewed-by: Martin Wilck <mwilck@suse.com>

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
mwilck pushed a commit to openSUSE/multipath-tools that referenced this issue Dec 2, 2021
This aligns 'multipathd' socket and service units, by adding the
start conditions that are set on the service but not on the socket.
It should help avoiding situations where the socket unit ends up
marked as failed after hitting its retry-limit.

Fixes: opensvc#15
Signed-off-by: Luca BRUNO <luca.bruno@coreos.com>
Reviewed-by: Martin Wilck <mwilck@suse.com>
bmarzins pushed a commit to bmarzins/rh-multipath-tools that referenced this issue Feb 17, 2022
Upstream Status: https://github.com/openSUSE/multipath-tools.git
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2008101
Conflicts: Match the conditions with RHEL-8 multipathd.service unit

commit 345ccf5
Author: Luca BRUNO <luca.bruno@coreos.com>
Date:   Fri Sep 24 09:34:01 2021 +0000

    multipathd.socket: add missing conditions from service unit

    This aligns 'multipathd' socket and service units, by adding the
    start conditions that are set on the service but not on the socket.
    It should help avoiding situations where the socket unit ends up
    marked as failed after hitting its retry-limit.

    Fixes: opensvc/multipath-tools#15
    Signed-off-by: Luca BRUNO <luca.bruno@coreos.com>
    Reviewed-by: Martin Wilck <mwilck@suse.com>

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
bmarzins pushed a commit to bmarzins/rh-multipath-tools that referenced this issue Feb 17, 2022
Upstream Status: https://github.com/openSUSE/multipath-tools.git
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2008101
Conflicts: Match the conditions with RHEL-8 multipathd.service unit

commit 345ccf5
Author: Luca BRUNO <luca.bruno@coreos.com>
Date:   Fri Sep 24 09:34:01 2021 +0000

    multipathd.socket: add missing conditions from service unit

    This aligns 'multipathd' socket and service units, by adding the
    start conditions that are set on the service but not on the socket.
    It should help avoiding situations where the socket unit ends up
    marked as failed after hitting its retry-limit.

    Fixes: opensvc/multipath-tools#15
    Signed-off-by: Luca BRUNO <luca.bruno@coreos.com>
    Reviewed-by: Martin Wilck <mwilck@suse.com>

Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants