container fails to start with: failed to mount shm tmpfs: invalid argument #14376

tmds · 2022-05-26T08:55:25Z

I have a container that runs fails to start with failed to mount shm ... : invalid argument.

The container used to start before.

# podman start gogs
Error: unable to start container "dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf": failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf/userdata/shm": invalid argument
[

Some info about the container:
It runs gogs: https://hub.docker.com/r/gogs/gogs.
One volume is mounted to an NFS folder. This is the /data volume, so probably not related to the error.
The container gets started as the root user on a Fedora 36 VM.

The text was updated successfully, but these errors were encountered:

rhatdan · 2022-05-26T09:44:04Z

If you remove and recreate the container does the problem go away?

Also could you get the --log-level=debug logs for this.

tmds · 2022-05-26T09:56:59Z

If you remove and recreate the container does the problem go away?

Yes. If I create a new container with the same arguments it starts successfully.

Also could you get the --log-level=debug logs for this.

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman start --log-level=debug gogs) 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /run/containers/storage       
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/libpod                    
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is being used 
DEBU[0000] Cached value indicated that native-diff is not being used 
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 7              
DEBU[0000] Made network namespace at /run/netns/netns-726e3a12-9b00-add4-e29b-ec0b093e3934 for container dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf 
DEBU[0000] Successfully loaded 1 networks               
[DEBUG netavark::network::validation] "Validating network namespace..."
[DEBUG netavark::commands::setup] "Setting up..."
[INFO  netavark::firewall] Using iptables firewall driver
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.ip_forward to 1
[DEBUG netavark::commands::setup] Setting up network podman with driver bridge
[DEBUG netavark::network::core] Container veth name: "eth0"
[DEBUG netavark::network::core] Brige name: "podman0"
[DEBUG netavark::network::core] IP address for veth vector: [10.88.0.23/16]
[DEBUG netavark::network::core] Gateway ip address vector: [10.88.0.1/16]
[DEBUG netavark::network::core] Configured static up address for eth0
[DEBUG netavark::network::core] Container veth mac: "9e:b4:26:a5:67:5e"
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-1D8721804F16F exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-1D8721804F16F exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK_FORWARD exists on table filter
[DEBUG netavark::network::core_utils] Setting sysctl value for net.ipv4.conf.podman0.route_localnet to 1
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-SETMARK exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-MASQ exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-DN-1D8721804F16F exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-DN-1D8721804F16F exists on table nat
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 10.88.0.0/16 -p tcp --dport 2022 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 10.88.0.0/16 -p tcp --dport 2022 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 127.0.0.1 -p tcp --dport 2022 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 127.0.0.1 -p tcp --dport 2022 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j DNAT -p tcp --to-destination 10.88.0.23:22 --destination-port 2022 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j DNAT -p tcp --to-destination 10.88.0.23:22 --destination-port 2022 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 10.88.0.0/16 -p tcp --dport 8082 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 10.88.0.0/16 -p tcp --dport 8082 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 127.0.0.1 -p tcp --dport 8082 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-HOSTPORT-SETMARK -s 127.0.0.1 -p tcp --dport 8082 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j DNAT -p tcp --to-destination 10.88.0.23:3000 --destination-port 8082 exists on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] rule -j DNAT -p tcp --to-destination 10.88.0.23:3000 --destination-port 8082 created on table nat and chain NETAVARK-DN-1D8721804F16F
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[DEBUG netavark::firewall::varktables::helpers] chain NETAVARK-HOSTPORT-DNAT exists on table nat
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-DN-1D8721804F16F -p tcp --dport 2022 -m comment --comment 'dnat name: podman id: dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf' exists on table nat and chain NETAVARK-HOSTPORT-DNAT
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-DN-1D8721804F16F -p tcp --dport 2022 -m comment --comment 'dnat name: podman id: dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf' created on table nat and chain NETAVARK-HOSTPORT-DNAT
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-DN-1D8721804F16F -p tcp --dport 8082 -m comment --comment 'dnat name: podman id: dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf' exists on table nat and chain NETAVARK-HOSTPORT-DNAT
[DEBUG netavark::firewall::varktables::helpers] rule -j NETAVARK-DN-1D8721804F16F -p tcp --dport 8082 -m comment --comment 'dnat name: podman id: dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf' created on table nat and chain NETAVARK-HOSTPORT-DNAT
[DEBUG netavark::commands::setup] {
        "podman": StatusBlock {
            dns_search_domains: Some(
                [],
            ),
            dns_server_ips: Some(
                [],
            ),
            interfaces: Some(
                {
                    "eth0": NetInterface {
                        mac_address: "9e:b4:26:a5:67:5e",
                        subnets: Some(
                            [
                                NetAddress {
                                    gateway: Some(
                                        10.88.0.1,
                                    ),
                                    ipnet: 10.88.0.23/16,
                                },
                            ],
                        ),
                    },
                },
            ),
        },
    }
[DEBUG netavark::commands::setup] "Setup complete"
DEBU[0001] Tearing down network namespace at /run/netns/netns-726e3a12-9b00-add4-e29b-ec0b093e3934 for container dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf 
[DEBUG netavark::commands::teardown] "Tearing down.."
[INFO  netavark::firewall] Using iptables firewall driver
[DEBUG netavark::commands::teardown] Setting up network podman with driver bridge
[DEBUG netavark::network::core_utils] bridge has 2 connected interfaces
[DEBUG netavark::network::core] Container veth name being removed: "eth0"
[DEBUG netavark::network::core] Container veth removed: "eth0"
[DEBUG netavark::commands::teardown] "Teardown complete"
DEBU[0001] Cleaning up container dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf 
DEBU[0001] failed to reset unit file: "Unit dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf.service not loaded." 
DEBU[0001] Network is already cleaned up, skipping...   
DEBU[0001] Container dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf storage is already unmounted, skipping... 
Error: unable to start container "dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf": failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf/userdata/shm": invalid argument

rhatdan · 2022-05-26T10:00:10Z

@mheon @vrothberg @giuseppe Ideas?

mheon · 2022-05-26T13:12:13Z

Any chance there's still a tmpfs mounted at the path Podman is trying to mount to? Maybe we failed to clean up the shm from the first time the container was run?

tmds · 2022-05-26T13:30:16Z

That isn't the case:

# mount | grep shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
shm on /var/lib/containers/storage/overlay-containers/173c3343234ee02f47d377903469abefdb14ebe454af223d7bad6b4864a3e534/userdata/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k,inode64)
shm on /var/lib/containers/storage/overlay-containers/e087f40c330f2b6df5fd61fd486f680ef5a4254bd7101d232aa12d39a69dec35/userdata/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k,inode64)
# podman start gogs
Error: unable to start container "dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf": failed to mount shm tmpfs "/var/lib/containers/storage/overlay-containers/dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf/userdata/shm": invalid argument

The invalid argument message may mean EINVAL is returned by some call during the mounting.

strace shows me this:

[pid 86433] mount("shm", "/var/lib/containers/storage/overlay-containers/dcc001e4a20629902cef7a73683d8efb7ceaef0863049a717e6c0b75e0d7cacf/userdata/shm", "tmpfs", MS_NOSUID|MS_NODEV|MS_NOEXEC, "mode=1777,size=65536000,context=\"system_u:object_r:container_file_t:s0:c724,c889\"") = -1 EINVAL (Invalid argument)

rhatdan · 2022-05-26T13:53:55Z

One potential idea.
Do you have SELinux enabled on your system? If not, you created this container before when SELinux was enabled and now are running with SELinux disabled.

tmds · 2022-05-26T13:59:23Z

That may be it. I disabled SELinux to see if it had to do with #14365.
I'll re-enable and see if I can start this container then.

rhatdan · 2022-05-26T14:13:09Z

Guaranteed that it what this is. Basically the context="..." is blowing on on a non SELinux machine.

tmds · 2022-05-26T14:26:00Z

Would it make sense for podman to try and mount without the context on EINVAL?

rhatdan · 2022-05-26T14:35:02Z

I am not sure what would be the correct thing to do here. The database says the container wants SELinux and SELinux is failing.

tmds · 2022-05-26T18:27:08Z

If SELinux is disabled on the system, the SELinux container configuration could be ignored instead of causing the container to stop working.

It's counter-intuitive (to me) that something stops working when SELinux gets disabled.

Anyway, now that I know what the cause is, I've re-created my container and it's working fine.

yrro · 2023-06-14T12:46:45Z

I just ran into this on Debian 12 ("bookworm"). I boot with security=selinux (but run with SELINUX=permissive). I have the same error from any podman run command:

Error: failed to mount shm tmpfs "/home/sam/.local/share/containers/storage/overlay-containers/32cdd71ace7e7f3b403282b61eaa7927197769d8f8f93370a5880cb3c4fa7e4a/userdata/shm": invalid argument

... and strace shows:

158282 mount("shm", "/home/sam/.local/share/containers/storage/overlay-containers/80adfe3bc131007456348885bfd5963e61b6c4e02dd63449c414c5ca5600682c/userdata/shm", "tmpfs", MS_NOSUID|MS_NODEV|MS_NOEXEC, "mode=1777,size=65536000,context=\"system_u:object_r:container_file_t:s0:c656,c1009\"") = -1 EINVAL (Invalid argument)

So far so identical to this issue. Except that I don't have SELinux disabled!

One additional detail is the kernel logging the message:

SELinux: security_context_str_to_sid (system_u:object_r:container_file_t:s0:c1022,c1023) failed with errno=-22

What I guess is going on is that Debian has not packaged container-selinux, so I guess the kernel has no idea what container_file_t is, and so the tmpfs mount with context= is rejected. I think this is shown by the fact that seinfo -t does not show any of the normal container_*_t types like it does on a Fedora system.

Does that previous paragraph make sense, if so I'll file a bug with Debian.

rhatdan · 2023-06-16T01:24:48Z

If you dont' have container-selinux installed SELinux separation will not work, and I don't believe that container-selinux will install without using fedora selinux-policy. So you would be best to remove /usr/share/containers/selinux/contexts

and see if this works, if it blows up then you might need to just set all the fields in that file to "", and see if that works. Otherwise you should disable SELinux separation in the containers.conf file.

yrro · 2023-06-16T09:03:11Z

Thanks for the input. Since container-selinux isn't easy to package for Debian I won't have a /usr/share/containers/selinux directory. So I've disabled labelling in containers.conf and podman works again! Thanks for that!

The only remaining mystery is how I was ever able to use podman on this system... I guess labelling could have been disabled in the past, and then a package update changed the default to true...

dulhaver · 2023-06-21T10:28:02Z

Thanks for the input. Since [container-selinux isn't easy to package for Debian(https://github.com/containers/container-selinux/issues/57) I won't have a /usr/share/containers/selinux directory. So I've disabled labelling in containers.conf and podman works again! Thanks for that!

for everybody who might be wondering how to practically do that and where:

sudo sed -i "s/#label = true/label = false/" /usr/share/containers/containers.conf

yrro · 2023-06-21T13:32:31Z

/usr/share/containers/containers.conf is owned by golang-github-containers-common and is not a conffile, so any changes made to this file will be undone when the package is upgraded.

Instead you just want to create the following file:

$ cat /etc/containers/containers.conf 
[containers]
label = false

See the man page containers.conf(5) for more details.

[additional]

Sorry Dan! But at least this way I can keep SELinux enabled overall on my system 😅

rhatdan · 2023-06-21T14:39:00Z

Which makes me cry. :^)
https://stopdisablingselinux.com/

yrro · 2023-06-21T17:05:57Z

Hold on a second. I just realised that refpolicy has its own container module these days!

I just built/installed, re-enabled container labelling, and Podman now works again!

(Hopefully this will contribute to preservation of Dan's tears...)

I don't know how refpolicy's module compares to container-selinux's module, but since it passed a smoke test I'll ask Debian to include it in selinux-policy-default.

abn0mad · 2023-06-29T15:46:29Z

@yrro - thank you for the work / discovery / post - you are a hero sir! With the impending demise of downstream RHEL based distributions, small organisations will have little choice but to switch to Debian + SELinux. I was so sad when I heard the news and tried to run a podman container on SELinux enabled Debian; tried to compile container-selinux but couldn't get it to work. This is the answer - my hat is of to thee good sir!

abn0mad · 2023-06-30T05:18:47Z

Update: it does not seem to work when booting with enforcing=1 on Debian 12 :(

Permission denied and no container_runtime_t on the conmon process, but container_t is active on the podman process.

Issue described here

rhatdan · 2023-06-30T10:47:18Z

What are the AVCs you are seeing, on the reference it cuts off the AVCs?

abn0mad · 2023-07-01T13:04:39Z

@rhatdan - apologies, I didn't copy the log correctly and deleted the VPS afterwards... I have since tried to recreate the error multiple times, but it seems to have "worked itself out" somehow, despite following the exact same steps and having checked, double-checked etc. I spun up a new VPS 4 times in an attempt to recreate the error.

I guess that means that Podman does now work with SELinux on on Debian 12, ~~but not with adequate security as conmon is unconfined.. so fixing that will be the next step...~~ - but it will need some tweaking and documenting

~~I will try to look at the pull request that brought the container policy to refpolicy and compare it with container-selinux and perhaps I can find a way to port some things over.~~

~~I'm not an SELinux specialist though, so it'll probably take me a while... On RHEL SELinux 'just works' - after one understands the basics, so I never got into policy development.~~

UPDATE: according to a refpolicy developer conmon is confined if run from systemd or a specific selinux role. Apparently Debian with SELinux does not assign roles to (system)users automatically. I shall do some more testing and report back.

rhatdan closed this as completed May 26, 2022

abn0mad mentioned this issue Jun 29, 2023

Allow container-selinux to be built on Debian containers/container-selinux#57

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Oct 1, 2023

github-actions bot locked as resolved and limited conversation to collaborators Oct 1, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

container fails to start with: failed to mount shm tmpfs: invalid argument #14376

container fails to start with: failed to mount shm tmpfs: invalid argument #14376

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

mheon commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

yrro commented Jun 14, 2023 •

edited

Loading

rhatdan commented Jun 16, 2023

yrro commented Jun 16, 2023 •

edited

Loading

dulhaver commented Jun 21, 2023 •

edited

Loading

yrro commented Jun 21, 2023 •

edited

Loading

rhatdan commented Jun 21, 2023

yrro commented Jun 21, 2023

abn0mad commented Jun 29, 2023

abn0mad commented Jun 30, 2023

rhatdan commented Jun 30, 2023

abn0mad commented Jul 1, 2023 •

edited

Loading

container fails to start with: failed to mount shm tmpfs: invalid argument #14376

container fails to start with: failed to mount shm tmpfs: invalid argument #14376

Comments

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

mheon commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

rhatdan commented May 26, 2022

tmds commented May 26, 2022

yrro commented Jun 14, 2023 • edited Loading

rhatdan commented Jun 16, 2023

yrro commented Jun 16, 2023 • edited Loading

dulhaver commented Jun 21, 2023 • edited Loading

yrro commented Jun 21, 2023 • edited Loading

rhatdan commented Jun 21, 2023

yrro commented Jun 21, 2023

abn0mad commented Jun 29, 2023

abn0mad commented Jun 30, 2023

rhatdan commented Jun 30, 2023

abn0mad commented Jul 1, 2023 • edited Loading

yrro commented Jun 14, 2023 •

edited

Loading

yrro commented Jun 16, 2023 •

edited

Loading

dulhaver commented Jun 21, 2023 •

edited

Loading

yrro commented Jun 21, 2023 •

edited

Loading

abn0mad commented Jul 1, 2023 •

edited

Loading