Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd status fails output on Ubuntu #8879

Open
2 tasks
elsaco opened this issue Sep 24, 2022 · 23 comments
Open
2 tasks

systemd status fails output on Ubuntu #8879

elsaco opened this issue Sep 24, 2022 · 23 comments

Comments

@elsaco
Copy link

elsaco commented Sep 24, 2022

Version

10.0.22621.521

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.15.62.1

Distro Version

Ubuntu 22.10

Other Software

No response

Repro Steps

install Ubuntu (Preview), store id 9P7BDVKVNXZ6. This version has systemd already enabled.

Expected Behavior

systemctl status command to work

Actual Behavior

toor@XPS:~$ systemctl status
Failed to dump process list for 'XPS', ignoring: Input/output error
● XPS
    State: degraded
    Units: 318 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 1 units
    Since: Sat 2022-09-24 08:43:15 PDT; 14min ago
  systemd: 251.4-1ubuntu6
  Tainted: cgroupsv1
   CGroup: /

It doesn't work on ArchLinux also:

[toor@XPS ~]$ systemctl status
Failed to dump process list for 'XPS', ignoring: Input/output error
● XPS
    State: running
    Units: 243 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2022-09-24 08:51:04 PDT; 14s ago
  systemd: 251.4-1-arch
  Tainted: cgroupsv1
   CGroup: /

Both WSL instances run systemd-251.4, so this could be a systemd issue. It works okay on Ubuntu 22.04 and systemd-249.11.

Diagnostic Logs

No response

@cerebrate
Copy link

Yeah, this one's been happening since systemd 250 shipped, under all the systemd hacks and now under the official support.

@Skepfyr
Copy link
Member

Skepfyr commented Sep 24, 2022

I've done some investigating:
This regression was introduced by systemd/systemd@038cae0, which comes from this PR systemd/systemd#21172. It's possible this is related to systemd/systemd#22089 but I'm not sure as it's slightly different symptoms and isn't fixed on main. That commit was a change to systemctl status to use the systemd dbus instead of "directly via cgroupfs" (I'm not sure what that means). I can replicate this bug by calling the dbus directly with busctl call -- org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager GetUnitProcesses 's' '-.slice'. Some code diving suggests that the cause of the IO error is that systemd is not expecting to see zero PIDs in /sys/fs/cgroup/unified/cgroup.procs, which I have a bunch of, I believe they're there because systemd is in a PID namespace but not in a new cgroup.

I see 3 possible places the bug might be:

  1. That systemctl status code: it possibly shouldn't be making that dbus call in that way, however, I'm skeptical as that API sounds sensible from its name.
  2. The systemd dbus handler for that method: systemd/systemd:src/core/dbus-unit.c:1310, specifically this line that generates the IO error systemd/systemd:src/basic/cgroup-util.c, that generates an error because of the previously mentioned zeros in /sys/fs/cgroup/unified/cgroup.procs.
  3. The WSL handling of cgroups: it's possible that systemd shouldn't observe zeros in its root cgroup. The systemd container interface suggests that systemd should be running in a new cgroup, but I don't know if that is required.

@cerebrate
Copy link

cerebrate commented Sep 25, 2022

I think it is 3; that's the same root cause @nullpo-head figured out for the problem when it was reported under distrod (nullpo-head/wsl-distrod#31 (comment)), with the same fix (put systemd in a new cgroup). Of course, with systemd now running as pid 1 without a hack, the fix'll need to be in WSL proper.

@cerebrate
Copy link

Still an issue on 0.68.2, FWIW.

@julianxhokaxhiu
Copy link

Still an issue also on the latest 1.0.0

$ wsl --version
Versione WSL: 1.0.0.0
Versione kernel: 5.15.74.2
Versione WSLg: 1.0.47
Versione MSRDC: 1.2.3575
Versione Direct3D: 1.606.4
Versione DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
versione Windows: 10.0.19045.2311

@BtbN
Copy link

BtbN commented Dec 13, 2022

I've applied the following workaround to my systemd on WSL, and it makes the output look like (I think) it's supposed to look:

diff -Naur systemd-stable-251.8.orig/src/core/dbus-unit.c systemd-stable-251.8/src/core/dbus-unit.c
--- systemd-stable-251.8.orig/src/core/dbus-unit.c      2022-12-13 23:27:56.133926327 +0100
+++ systemd-stable-251.8/src/core/dbus-unit.c   2022-12-13 23:39:03.772622555 +0100
@@ -1288,6 +1288,8 @@
                  * “cgroup.procs” in a threaded domain cgroup contains the PIDs of all processes in
                  * the subtree and is not readable in the subtree proper. */
                 r = cg_read_pid(f, &pid);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

It's most likely not a proper fix or a good idea to do, but it gets stuff working for me in the meantime.

@chutzimir
Copy link

I've applied the following workaround to my systemd on WSL, and it makes the output look like (I think) it's supposed to look:

diff -Naur systemd-stable-251.8.orig/src/core/dbus-unit.c systemd-stable-251.8/src/core/dbus-unit.c
--- systemd-stable-251.8.orig/src/core/dbus-unit.c      2022-12-13 23:27:56.133926327 +0100
+++ systemd-stable-251.8/src/core/dbus-unit.c   2022-12-13 23:39:03.772622555 +0100
@@ -1288,6 +1288,8 @@
                  * “cgroup.procs” in a threaded domain cgroup contains the PIDs of all processes in
                  * the subtree and is not readable in the subtree proper. */
                 r = cg_read_pid(f, &pid);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

It's most likely not a proper fix or a good idea to do, but it gets stuff working for me in the meantime.

Thanks. I also rebuilt the systemd-250-12.el9_1 SRPM on Rocky 9.1 with a patch similar to what you showed above. It will do the trick until it gets fixed somewhere.

This is on

> wsl --version
WSL version: 1.0.3.0
Kernel version: 5.15.79.1
WSLg version: 1.0.47
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22000.1335

@allenatdecisiv
Copy link

This is also happening on Fedora 36:

❯ sudo systemctl status
Failed to dump process list for 'fedoraremix', ignoring: Input/output error
● fedoraremix
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2023-01-30 13:33:55 EST; 2min 58s ago
   CGroup: /

systemd version:

❯ sudo systemctl --version
systemd 250 (v250.9-1.fc36)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

WSL version:

❯ wsl --version
WSL version: 1.0.3.0
Kernel version: 5.15.79.1
WSLg version: 1.0.47
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.2546

@mokeyish
Copy link

This is also happening on Kali-linux

@MorphBonehunter
Copy link

I guess this issue will get more attention over time with more distos update systemd to >= 250. At the moment it seems that the only supported distro with such an systemd version is Kali (but I did not verify this).

@allenatdecisiv
Copy link

@MorphBonehunter,

The patch provided by @BtbN does work for me on Fedora 36 using the technique described by @chutzimir.

❯ sudo systemctl status
● fedoraremix
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2023-03-01 15:11:12 EST; 5min ago
   CGroup: /
           ├─   2 /init
           ├─   5 plan9 --control-socket 6 --log-level 4 --server-fd 7 --pipe-fd 9 --log-truncate
           ├─ 243 /init
           ├─ 255 /bin/sh /usr/lib/bottle-imp/wait-forever.sh
           ├─ 257 sleep 1h
           ├─ 456 /init
           ├─ 457 /init
           ├─ 458 machinectl -E WT_SESSION=20dace65-44b2-4ed6-9001-7429bf8779d5 -E WT_PROFILE_ID={d8637476-55e3-42dd-ab39-3f25bc643>
           ├─init.scope
           │ └─ 1 /sbin/init
           ├─system. Slice
❯ systemctl --version
systemd 250 (v250.10-2.1.fc36)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
❯ wsl --version
WSL version: 1.0.3.0
Kernel version: 5.15.79.1
WSLg version: 1.0.47
MSRDC version: 1.2.3575
Direct3D version: 1.606.4
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.2673

@elsaco
Copy link
Author

elsaco commented Mar 1, 2023

Using different systemd versions systemctl status works sometimes.

openSUSE Tumbleweed with systemd 252.5, not working:

elsaco@eleven:~> systemctl status
Failed to dump process list for 'eleven', ignoring: Input/output error
● eleven
    State: running
    Units: 289 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2023-03-01 12:13:31 PST; 21min ago
  systemd: 252.5+suse.46.gd87834a334
  Tainted: cgroupsv1
   CGroup: /

Ubuntu 22.04 with systemd 249.11, working:

elsaco@eleven:~$ systemctl status --no-pager
● eleven
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2023-03-01 12:35:34 PST; 22s ago
   CGroup: /
           ├─docker
           │ └─efdf49cbafdcccda6b8ce5077932766b38a74d0d41af44dbe4fe4d36f50a28d6
           ├─user.slice
           │ └─user-1000.slice
           │   ├─user@1000.service
           │   │ └─init.scope
           │   └─session-c1.scope
           ├─init.scope
           └─system.slice
---cut---

Both distros run on:

WSL version: 1.1.3.0
Kernel version: 5.15.90.1
WSLg version: 1.0.49
MSRDC version: 1.2.3770
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.1344

@SuperSandro2000
Copy link

I've applied the following workaround to my systemd on WSL, and it makes the output look like (I think) it's supposed to look:

diff -Naur systemd-stable-251.8.orig/src/core/dbus-unit.c systemd-stable-251.8/src/core/dbus-unit.c
--- systemd-stable-251.8.orig/src/core/dbus-unit.c      2022-12-13 23:27:56.133926327 +0100
+++ systemd-stable-251.8/src/core/dbus-unit.c   2022-12-13 23:39:03.772622555 +0100
@@ -1288,6 +1288,8 @@
                  * “cgroup.procs” in a threaded domain cgroup contains the PIDs of all processes in
                  * the subtree and is not readable in the subtree proper. */
                 r = cg_read_pid(f, &pid);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

It's most likely not a proper fix or a good idea to do, but it gets stuff working for me in the meantime.

Works and was dead easy to get into NixOS:

nixpkgs.overlays = [
  (final: prev: with final; {
    systemd-wsl = final.systemd.overrideAttrs ({ patches, ... }: {
      patches = patches ++ [
        ./systemd-systemctl-status-wsl.patch
      ];
    });
  })
];

systemd.package = pkgs.systemd-wsl;

@CarlosNihelton
Copy link
Contributor

Still an issue to date.

Versão do WSL: 1.2.0.0
Versão do kernel: 5.15.90.1
Versão do WSLg: 1.0.51
Versão do MSRDC: 1.2.3770
Versão do Direct3D: 1.608.2-61064218
Versão do DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Versão do Windows: 10.0.25336.1010

Also tested with pre-release 1.2.4. Both using cgroups v2. Of the Ubuntu apps, that situation still only affects the Ubuntu-Preview for now, but eventually other releases will be at systemd version 2.50 or later, so that will be a problem.

A suitable fix seems to be to create and assign a cgroup and a cgroup namespace to every distro instance. Handling the cgroup2 mount point properly before starting systemd, it should then see that as its root cgroup. There will be no PID 0's inside the cgroup.procs file, because it will be populated only by the PIDs of the child processes of systemd. Also, care must be taken to ensure the 'login" shell also goes to the that cgroup (and namespaces).

For the sake of demonstration, if instead of enabling systemd through the proper settings in wsl.conf we use the hack described below, then we get systemctl status behaving correctly, with the downside of the root cgroup being displayed not as /.

An extension to an older hack script to start systemd in its own namespace and cgroup.

#!/bin/sh
set -eu

# Ensure that we don't clean up /tmp setup by the Microsoft init process.
mkdir -p /run/tmpfiles.d
ln -s /dev/null /run/tmpfiles.d/tmp.conf

umount /proc/sys/fs/binfmt_misc

# Create the distro cgroup
mkdir -p /sys/fs/cgroup/distros/ubuntu-preview

# Start systemd in its own namespace. Notice the `-C` for cgroup
env -i /usr/bin/unshare -Cm --fork --mount-proc --pid --propagation unchanged -- sh -c '
echo $PPID > /sys/fs/cgroup/distros/ubuntu-preview/cgroup.procs && exec /lib/systemd/systemd --unit=multi-user.target
' &

DISCLAIMER: By no means the above is a solution. It has many drawbacks, such as not showing the systemd root cgroup as / as well as requiring more hacks to run processes (such as a shell) under the systemd namespaces.

Configuring the boot command parameter to run that script and starting a shell under systemd namespaces allows us to see the following systemctl output.

/etc/wsl.conf

[boot]
command=<absolute path of the script>
carlos@Zero01:~$ systemctl status
● Zero01
    State: running
    Units: 312 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Wed 2023-04-19 15:18:12 -03; 5s ago
  systemd: 252.5-2ubuntu3
   CGroup: /distros/ubuntu-preview
           ├─init.scope
           │ └─1 /lib/systemd/systemd --unit=multi-user.target
           └─system.slice
             ├─console-getty.service
             │ └─125 /sbin/agetty -o "-p -- \\u" --noclear --keep-baud - 115200,38400,9600 vt220
             ├─cron.service
             │ └─111 /usr/sbin/cron -f -P
             ├─dbus.service
             │ └─112 @dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
             ├─rsyslog.service
             │ └─124 /usr/sbin/rsyslogd -n -iNONE
             ├─snap-core22-607.mount
             │ └─62 snapfuse /var/lib/snapd/snaps/core22_607.snap /snap/core22/607 -o ro,nodev,allow_other,suid
             ├─snap-snapd-18596.mount
             │ └─63 snapfuse /var/lib/snapd/snaps/snapd_18596.snap /snap/snapd/18596 -o ro,nodev,allow_other,suid
             ├─snap-snapd-18933.mount
             │ └─65 snapfuse /var/lib/snapd/snaps/snapd_18933.snap /snap/snapd/18933 -o ro,nodev,allow_other,suid
             ├─snap-subiquity-4692.mount
             │ └─64 snapfuse /var/lib/snapd/snaps/subiquity_4692.snap /snap/subiquity/4692 -o ro,nodev,allow_other,suid
             ├─snap-subiquity-4714.mount
             │ └─61 snapfuse /var/lib/snapd/snaps/subiquity_4714.snap /snap/subiquity/4714 -o ro,nodev,allow_other,suid
             ├─snap.subiquity.subiquity-server.service
             │ ├─117 /bin/sh /snap/subiquity/4714/usr/bin/subiquity-server
             │ └─186 /snap/subiquity/4714/usr/bin/python3.10 -m subiquity.cmd.server
             ├─snapd.service
             │ └─119 /usr/lib/snapd/snapd

@xiaoyang-sde
Copy link

This issue is reproducible on WSL-based Arch Linux with systemd 254 (254.5-1-arch).

$ sudo SYSTEMD_LOG_LEVEL=7 SYSTEMD_PAGER= systemctl status
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a
● MSI-B660M
    State: running
    Units: 252 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2023-11-09 11:18:50 PST; 25min ago
  systemd: 254.5-1-arch
  Tainted: cgroupsv1
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=2 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.IOError error-message=Input/output error
Failed to dump process list for 'MSI-B660M', ignoring: Input/output error
Bus n/a: changing state RUNNING → CLOSED

@Splarv
Copy link

Splarv commented Dec 14, 2023

The same with debian

C:\Users\splar>wsl --version
Версия WSL: 2.0.9.0
Версия ядра: 5.15.133.1-1
Версия WSLg: 1.0.59
Версия MSRDC: 1.2.4677
Версия Direct3D: 1.611.1-81528511
Версия DXCore: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Версия Windows: 10.0.19045.3803

root@dog:~# dpkg -s systemd
Package: systemd
Status: install ok installed
Priority: important
Section: admin
Installed-Size: 9647
Maintainer: Debian systemd Maintainers <pkg-systemd-maintainers@lists.alioth.debian.org>
Architecture: amd64
Multi-Arch: foreign
Version: 252.19-1~deb12u1

root@dog:~# systemctl status
Failed to dump process list for 'dog', ignoring: Input/output error
● dog
    State: running
    Units: 300 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Thu 2023-12-14 23:36:50 MSK; 17min ago
  systemd: 252.19-1~deb12u1
  Tainted: cgroupsv1
   CGroup: /

@Tatsh
Copy link

Tatsh commented Jan 8, 2024

Gentoo.

WSL version: 2.0.9.0
Kernel version: 5.15.133.1-1
WSLg version: 1.0.59
MSRDC version: 1.2.4677
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.19045.3693
# uname -a
Linux WX-5CG3342PM9 6.7.0-gentoo-microsoft-standard-WSL2 #3 SMP Mon Jan  8 14:07:42 EST 2024 x86_64 12th Gen Intel(R) Core(TM) i7-1265U GenuineIntel GNU/Linux
# systemctl status
Failed to dump process list for '...', ignoring: Input/output error
● WX-5CG3342PM9
    State: running
    Units: 258 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2024-01-08 14:58:00 EST; 56s ago
  systemd: 255
  Tainted: cgroupsv1
   CGroup: /

@PaulKGrimes
Copy link

PaulKGrimes commented Feb 13, 2024

Still an issue with Oracle Linux 9.2 installed and updated from the Microsoft Store:

WSL version: 2.0.14.0
Kernel version: 5.15.133.1-1
WSLg version: 1.0.59
MSRDC version: 1.2.4677
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22621.3007
systemctl --version
systemd 252 (252-18.0.1.el9)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
sudo systemctl status
Failed to dump process list for 'SI388844-Oracle9', ignoring: Input/output error
● SI388844-Oracle9
    State: running
    Units: 137 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Tue 2024-02-13 11:34:55 EST; 33s ago
  systemd: 252-18.0.1.el9
   CGroup: /

@Tatsh
Copy link

Tatsh commented Feb 27, 2024

Updated patch for v255:

diff --git a/src/core/dbus-unit.c b/src/core/dbus-unit.c
index 1a037b7..f0550e8 100644
--- a/src/core/dbus-unit.c
+++ b/src/core/dbus-unit.c
@@ -1300,6 +1300,8 @@ static int append_cgroup(sd_bus_message *reply, const char *p, Set *pids) {
                  * readable in the subtree proper. */

                 r = cg_read_pidref(f, &pidref);
+                if (r == -EIO)
+                        continue;
                 if (IN_SET(r, 0, -EOPNOTSUPP))
                         break;
                 if (r < 0)

@trallnag
Copy link

trallnag commented Apr 25, 2024

Also seeing this.

> wsl --version
WSL-Version: 2.1.5.0
Kernelversion: 5.15.146.1-2
WSLg-Version: 1.0.60
MSRDC-Version: 1.2.5105
Direct3D-Version: 1.611.1-81528511
DXCore-Version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows-Version: 10.0.22631.3447
> systemctl --version
systemd 252 (252.22-1~deb12u1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
❯ PAGER= SYSTEMD_LOG_LEVEL=debug systemctl status
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a
● metro
    State: degraded
    Units: 282 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 2 units
    Since: Thu 2024-04-25 23:12:44 CEST; 16min ago
  systemd: 252.22-1~deb12u1
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=2 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.IOError error-message=Input/output error
Failed to dump process list for 'metro', ignoring: Input/output error
Bus n/a: changing state RUNNING → CLOSED

@trallnag
Copy link

I guess this issue will get more attention over time with more distos update systemd to >= 250. At the moment it seems that the only supported distro with such an systemd version is Kali (but I did not verify this).

@MorphBonehunter, yep. Ubuntu 24 is around the corner and it uses systemd 255

@Stanzilla
Copy link

Yes, do not update to Ubuntu 24.04, I made that mistake yesterday.

https://twitter.com/stanzillaz/status/1783274242995290244

@yveshield
Copy link

yveshield commented May 1, 2024

This issue is reproducible on WSL-based Ubuntu Linux with systemd 253.5-1ubuntu6.1.

$ sudo PAGER= SYSTEMD_LOG_LEVEL=debug systemctl status
Bus n/a: changing state UNSET → OPENING
sd-bus: starting bus by connecting to /run/systemd/private...
Bus n/a: changing state OPENING → AUTHENTICATING
Bus n/a: changing state AUTHENTICATING → RUNNING
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a
● wsl
    State: degraded
    Units: 368 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 2 units
    Since: Wed 2024-05-01 14:33:38 CST; 2h 21min ago
  systemd: 253.5-1ubuntu6.1
  Tainted: cgroupsv1
   CGroup: /
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnitProcesses cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=error sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a  cookie=2 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.IOError error-message=Input/output error
Failed to dump process list for 'wsl', ignoring: Input/output error
Bus n/a: changing state RUNNING → CLOSED```

keszybz added a commit to keszybz/systemd that referenced this issue May 8, 2024
In some environments, namely WSL2, the cgroup.procs PID list for some reason
contain a ton of zeros everywhere, most likely those are from other instances
under the same WSL Kernel, which at least always hosts the system instance with
the X/Wayland/PA/Pipe server.

Without this patch, whenever cg_read_pid encounters such a zero, it throws an
error. This makes systemd near unusable inside of WSL. Change cg_read_pid()
to return 0, and adjust all callers to handle that appropriately. In general,
we cannot do anything with such processes, so most operations have to be refused.
The only thing we can do with them is count them, in particular, we can answer
the question whether the cgroup is empty in the negative.

On normal systems, where the list does not contain any zeros to begin with,
this has no averse effects.

See also: microsoft/WSL#8879.

Co-authored-by: Timo Rothenpieler <timo@rothenpieler.org>
keszybz added a commit to keszybz/systemd that referenced this issue May 8, 2024
In some environments, namely WSL2, the cgroup.procs PID list for some reason
contain a ton of zeros everywhere, most likely those are from other instances
under the same WSL Kernel, which at least always hosts the system instance with
the X/Wayland/PA/Pipe server.

Without this patch, whenever cg_read_pid encounters such a zero, it throws an
error. This makes systemd near unusable inside of WSL. Change cg_read_pid()
to return 0, and adjust all callers to handle that appropriately. In general,
we cannot do anything with such processes, so most operations have to be refused.
The only thing we can do with them is count them, in particular, we can answer
the question whether the cgroup is empty in the negative.

On normal systems, where the list does not contain any zeros to begin with,
this has no averse effects.

This replaces systemd#32534.
See also: microsoft/WSL#8879.

Co-authored-by: Timo Rothenpieler <timo@rothenpieler.org>
keszybz added a commit to keszybz/systemd that referenced this issue May 8, 2024
In some environments, namely WSL2, the cgroup.procs PID list for some reason
contain a ton of zeros everywhere, most likely those are from other instances
under the same WSL Kernel, which at least always hosts the system instance with
the X/Wayland/PA/Pipe server.

Without this patch, whenever cg_read_pid encounters such a zero, it throws an
error. This makes systemd near unusable inside of WSL. Change cg_read_pid()
to return 0, and adjust all callers to handle that appropriately. In general,
we cannot do anything with such processes, so most operations have to be refused.
The only thing we can do with them is count them, in particular, we can answer
the question whether the cgroup is empty in the negative.

On normal systems, where the list does not contain any zeros to begin with,
this has no averse effects.

This replaces systemd#32534.
See also: microsoft/WSL#8879.

Co-authored-by: Timo Rothenpieler <timo@rothenpieler.org>
keszybz added a commit to keszybz/systemd that referenced this issue May 9, 2024
In some environments, namely WSL2, the cgroup.procs PID list for some reason
contain a ton of zeros everywhere, most likely those are from other instances
under the same WSL Kernel, which at least always hosts the system instance with
the X/Wayland/PA/Pipe server.

Without this patch, whenever cg_read_pid encounters such a zero, it throws an
error. This makes systemd near unusable inside of WSL. Change cg_read_pid()
to return 0, and adjust all callers to handle that appropriately. In general,
we cannot do anything with such processes, so most operations have to be refused.
The only thing we can do with them is count them, in particular, we can answer
the question whether the cgroup is empty in the negative.

On normal systems, where the list does not contain any zeros to begin with,
this has no averse effects.

This replaces systemd#32534.
See also: microsoft/WSL#8879.

Co-authored-by: Timo Rothenpieler <timo@rothenpieler.org>
keszybz added a commit to keszybz/systemd that referenced this issue May 9, 2024
In some environments, namely WSL2, the cgroup.procs PID list for some reason
contain a ton of zeros everywhere, most likely those are from other instances
under the same WSL Kernel, which at least always hosts the system instance with
the X/Wayland/PA/Pipe server.

Without this patch, whenever cg_read_pid encounters such a zero, it throws an
error. This makes systemd near unusable inside of WSL. Change cg_read_pid()
to return 0, and adjust all callers to handle that appropriately. In general,
we cannot do anything with such processes, so most operations have to be refused.
The only thing we can do with them is count them, in particular, we can answer
the question whether the cgroup is empty in the negative.

On normal systems, where the list does not contain any zeros to begin with,
this has no averse effects.

This replaces systemd#32534.
See also: microsoft/WSL#8879.

Co-authored-by: Timo Rothenpieler <timo@rothenpieler.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests