ubuntu+systemd: Failed to mount tmpfs at /run/lock: Operation not permitted

[space88man]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug
Description
Ubuntu 19.04 + systemd container is failing with

Failed to mount tmpfs at /run/lock: Operation not permitted

This happens on one Fedora 30 host, but another one runs the container with no issue.
--log-level DEBUG and journalctl output are not sufficiently illuminating to explain why this
would work on one host but not the other.

Steps to reproduce the issue:

1. Create ubuntu+system container

CON=$(buildah from ubuntu:19.04)
buildah run $CON apt update
buildah run $CON apt install -y systemd systemd-sysv
buildah run $CON systemctl mask systemd-resolved.service
buildah commit $CON ubuntu:local

2. Run the container:

podman run -it -a stdin,stdout,stderr --rm --name test_1 --entrypoint /sbin/init --stop-signal rtmin+3 \
    --env container=podman ubuntu:local

Describe the results you received:

DEBU[0000] Created container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b in OCI
 runtime 
DEBU[0000] Attaching to container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b 
DEBU[0000] connecting to socket /var/run/libpod/socket/e87ddccce671077c552318383647ec81b15480f4fefbd
ce69088e2522a0ea75b/attach 
DEBU[0000] Received a resize event: {Width:100 Height:47} 
DEBU[0000] Starting container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b with 
command [/sbin/init] 
DEBU[0000] Started container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b 
Failed to mount tmpfs at /run/lock: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
DEBU[0000] Enabling signal proxying                     
DEBU[0000] Checking container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b statu
s... 
DEBU[0000] Cleaning up container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b 
DEBU[0000] Tearing down network namespace at /var/run/netns/cni-ff28cbe6-2d57-c40b-ce08-9bb0353f6e7f
 for container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b 
INFO[0000] Got pod network &{Name:test_1 Namespace:test_1 ID:e87ddccce671077c552318383647ec81b15480f
4fefbdce69088e2522a0ea75b NetNS:/var/run/netns/cni-ff28cbe6-2d57-c40b-ce08-9bb0353f6e7f PortMappings
:[] Networks:[] NetworkConfig:map[]} 
INFO[0000] About to del CNI network podman (type=bridge) 
DEBU[0000] unmounted container "e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b" 
DEBU[0000] Successfully cleaned up container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e25
22a0ea75b 
DEBU[0000] Container e87ddccce671077c552318383647ec81b15480f4fefbdce69088e2522a0ea75b storage is alr
eady unmounted, skipping... 

Describe the results you expected:
Container runs

Additional information you deem important (e.g. issue happens only occasionally):

• Host: Fedora 30; this happens on one F30 host, and another has no issue.
• Same result with debian:9 instead of ubuntu:19.04. Again this manifests itself on one F30 host
  and another F30 host works fine.
• The problematic host runs fedora:30+systemd and centos:7+systemd without any issues

Output of podman version:

# podman version
Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:                                                                                              
  compiler: gc                                                                                      
  git commit: ""                                                                                    
  go version: go1.12.2                                                                              
  podman version: 1.3.1                                                                             
host:                                                                                               
  BuildahVersion: 1.8.2                                                                             
  Conmon:                                                                                           
    package: podman-1.3.1-1.git7210727.fc30.x86_64                                                  
    path: /usr/libexec/podman/conmon                                                                
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'          
  Distribution:                                                                                     
    distribution: fedora                    
    version: "30"
  MemFree: 26918481920
  MemTotal: 33399087104
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 8
  hostname: podman.localhost
  kernel: 5.1.7-300.fc30.x86_64
  os: linux
  rootless: false
  uptime: 21h 41m 54.08s (Approximately 0.88 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 9
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

[space88man]

Failing host:

DEBU[0000] Starting container a0a4f3cf47be7e9e29b9887f485f1e56b9c1b909280266cfc540619450287acc with command [/sbin/init] 
DEBU[0000] Started container a0a4f3cf47be7e9e29b9887f485f1e56b9c1b909280266cfc540619450287acc 
Failed to mount tmpfs at /run/lock: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

Successful host:

DEBU[0002] Starting container 0bfa5f151d555b8549b9acfbed1bb046305725bf1aaa47fe01da833a14748e13 with command [/sbin/init] 
DEBU[0002] Started container 0bfa5f151d555b8549b9acfbed1bb046305725bf1aaa47fe01da833a14748e13 
systemd 240 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +L
Z4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
Detected virtualization container-other.
Detected architecture x86-64.

Welcome to Ubuntu 19.04!

Set hostname to <0bfa5f151d55>.

[space88man]

Update: the problem host was missing oci-systemd-hook. With this package installed the host now runs the container correctly.

I vaguely remember reading somewhere that this package was no longer required and that podman and runc together do the systemd dance. I won't close the issue immediately in case the maintainers want to chime in on oci-systemd-hook necessity.

Both fedora:30 and centos:7 work without this package, but ubuntu:19.04 and debian:9 seem to require it.

[rhatdan]

Ok then this looks like a bug in either systemd inside of a container or the implementation of podman.

[mheon]

Hm. systemd is trying to do an explicit mount of /run/lock as a tmpfs... but I seem to recall being told it didn't require that, so long as all of /run was a tmpfs... Hm.

[rhatdan]

The issue is that if /run/lock is not mounted separately, then systemd will attempt to mount it. Since mount is blocked then systemd fails. Bottom line SYSTEMD expects /run/lock to be a separate mount point.

[LQss11]

I am not sure if this helps or not but I got the same error trying to start a docker container and boot it with systemd as init system (PID 1) and solved that adding --privileged tag like so

docker run -ti -d --privileged ubuntu:20.04 "/sbin/init"

[rhatdan]

Run it with Podman and you have no requirement for --privileged flag.