flake: ubuntu 2010: systemd: Failed to create ... control group, EPERM

[edsantiago]

Starting to see this flake often on ubuntu:

$ podman run ... -t -i -d registry.access.redhat.com/ubi8-init /sbin/init
systemd 239 (239-41.el8_3.2) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Red Hat Enterprise Linux 8.3 (Ootpa)

Set hostname to <7ee143227d95>.
Initializing machine ID from random generator.
Failed to read AF_UNIX datagram queue length, ignoring: No such file or directory
Failed to install release agent, ignoring: Permission denied
Failed to create /user.slice/user-27341.slice/session-3.scope/init.scope control group: Permission denied
Failed to allocate manager object: Permission denied
!!!!!!
Failed to allocate manager object, freezing.
Freezing execution.

Podman systemd [It] podman run container with systemd PID1

• ubuntu-2010 : int podman ubuntu-2010 rootless host
  • PR Fix formatting and indentation in network http api docs #10369
    • 05-17 17:26
    • 05-17 17:26
    • 05-17 17:26
  • PR Several shell completion fixes #10328
    • 05-17 10:40
    • 05-17 10:40
    • 05-17 10:40
  • PR Add host.serviceIsRemote to podman info results #10300
    • 05-10 18:01
    • 05-10 18:01
    • 05-10 18:01
  • PR Remove obsolete skips #10291
    • 05-10 10:29
    • 05-10 10:29
    • 05-10 10:29
  • PR Update nix pin with make nixpkgs #10239
    • 05-11 22:06
    • 05-11 22:06
    • 05-11 22:06

[rhatdan]

Well it is not SELinux, is there anyway we could check the audit.log to see if there is any seccomp failures?

[edsantiago]

OMG yes there is! Thanks to the foresight of @cevich, audit.log is preserved! For any of the above logs, go to the top, click the Task link, scroll to the bottom, and expand the audit accordion. Example. (I have no idea what to look for, though)

[rhatdan]

No seccomp failures, that I see.

[rhatdan]

This looks like ubuntu occasionally has a cgroup file system that is not allowed to be written from within the container.
@giuseppe Any thoughts?

[giuseppe]

cgroup v1 for systemd doesn't need to be writeable, just /sys/fs/cgroup/systemd.

Are the rootless tests running from a systemd-run --scope --user environment?

[edsantiago]

I don't know what that means. They are running as a user with the magic loginctl enabled, via ansible become_user.

[edsantiago]

Oops - correction, these are failing in CI, not gating tests. I don't know what the setup is for rootless (wrt loginctl) but AFAICT those are run via ssh user@localhost. The interesting thing is that this is e2e tests, not system tests. e2e tests do automatic triple-retries, and in all the cases above, all three fail. That is: once a host gets into a mode where this test fails, it will always fail. A CI retry (which spins up a new VM) then succeeds.

[cevich]

Weird.

AFAICT those are run via ssh user@localhost

Correct.

e2e tests do automatic triple-retries

Additionally: The tests run in a randomized order. The "seed" for the order is displayed at the beginning. It's possible to run the tests with a specified seed, to guarantee the order. It might be useful to nail-down whether or not this failure is influenced by another test munging the system, or is the failure ever reproducible when running just this one test in isolation (on a fresh VM)

[edsantiago]

Podman systemd [It] podman run container with systemd PID1

• ubuntu-2010 : int podman ubuntu-2010 rootless host
  • PR generate systemd: make mounts portable #10506
    • 05-31 06:23
    • 05-31 06:23
    • 05-31 06:23
  • PR Add options to podman machine ssh #10485
    • 05-27 11:01
    • 05-27 11:01
    • 05-27 11:01
  • PR Fix formatting and indentation in network http api docs #10369
    • 05-17 17:26
    • 05-17 17:26
    • 05-17 17:26
  • PR Several shell completion fixes #10328
    • 05-17 10:40
    • 05-17 10:40
    • 05-17 10:40

[edsantiago]

Podman systemd [It] podman run container with systemd PID1

• ubuntu-2010 : int podman ubuntu-2010 rootless host
  • PR [v3.2] System tests: add :Z to volume mounts #10532
    • 06-02 09:05
    • 06-02 09:05
    • 06-02 09:05
  • PR [v3.2] generate systemd: make mounts portable #10509
    • 05-31 09:46
    • 05-31 09:46
    • 05-31 09:46

[rhatdan]

Could this be AppArmor causing this? Has anyone looked at the audit.log to see if apparmor is complaining about something?

[edsantiago]

For any log link above: click the link, Press Home key to go to the top of the log, click the Task link at top. That takes you to the Cirrus page for the failed run. I don't see an AppArmor log anywhere, but there's a Run journal tab that includes things like "PID", "scope", "control group", and EPERM.