Set up /media and /mnt to match the host

[Zlopez]

Today I suddenly couldn't enter my toolbox, so I tried to recreate it again using toolbox reset and toolbox enter with no luck. Here is the output of toolbox -v enter:

toolbox: running as real user ID 1000
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: checking if /etc/subgid and /etc/subuid have entries for user zlopez
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: running on a cgroups v2 host
toolbox: current Podman version is 1.6.2
toolbox: migration not needed: Podman version 1.6.2 is unchanged
toolbox: Fedora generational core is f31
toolbox: base image is fedora-toolbox:31
toolbox: container is fedora-toolbox-31
toolbox: checking if container fedora-toolbox-31 exists
toolbox: calling org.freedesktop.Flatpak.SessionHelper.RequestSession
toolbox: starting container fedora-toolbox-31
toolbox: /etc/profile.d/toolbox.sh already mounted in container fedora-toolbox-31
Error: unable to start container "fedora-toolbox-31": creating file '/var/home/zlopez/.local/share/containers/storage/overlay/bcd97a238cf639f8d3dfeef7b5c44b7ad9f4ba99410864856358b26ade201f1e/merged/media': Is a directory: OCI runtime error
toolbox: failed to start container fedora-toolbox-31

[elhutchi]

I had the same issue after upgrading Silverblue. These were the packages included in the upgrade. It was fixed after rolling back.

       Upgraded: crun 0.10.2-1.fc31 -> 0.10.5-2.fc31
                 kernel 5.3.8-300.fc31 -> 5.3.9-300.fc31
                 kernel-core 5.3.8-300.fc31 -> 5.3.9-300.fc31
                 kernel-devel 5.3.8-300.fc31 -> 5.3.9-300.fc31
                 kernel-modules 5.3.8-300.fc31 -> 5.3.9-300.fc31
                 kernel-modules-extra 5.3.8-300.fc31 -> 5.3.9-300.fc31

[Zlopez]

Probably a new version of crun caused this.

[garrett]

I have an old spare laptop I use for testing; it's running Silverblue Rawhide. Toolbox is working there on its most recent update (Rawhide.20191107.n.1), but it's between the F31 versions: crun-0.10.4-1.fc32.x86_64.

So it seems something broke in crun between 0.10.4 and 0.10.5. Hopefully this helps to isolate the problem and either fix it in crun or podman... or work around or adapt to it in toolbox.

However, I should note that podman in rawhide is at version 1.6.3-0.34.dev.git1e750f7.fc32.x86_64, not podman-1.6.2-2.fc31.x86_64.

Meanwhile, toolbox is on the same version in all the places, toolbox-0.0.16-1.fc31.noarch (or toolbox-0.0.16-1.fc32.noarch in rawhide).

[giuseppe]

would you please share the output of stat /media from the host?

[giuseppe]

I've opened a PR to fix it in crun. In the meanwhile, you can workaround it with:

diff --git a/toolbox b/toolbox
index a7433e1..6b443c7 100755
--- a/toolbox
+++ b/toolbox
@@ -987,7 +987,7 @@ create()
     fi
 
     if [ -d /run/media ] 2>&3; then
-        run_media_path_bind="--volume /run/media:/run/media:rslave"
+        run_media_path_bind="--volume $(readlink -f /run/media):/run/media:rslave"
     fi
 
     echo "$base_toolbox_command: checking if /usr is mounted read-only or read-write" >&3

[HarryMichal]

Thank you for the quick fix, @giuseppe!!

[p1u3o]

I'm having a similar issue with /mnt

Error: unable to start container "fedora-toolbox-31": creating file '/var/home/pluto/.local/share/containers/storage/overlay/e56a2816dbb492d3446030ba65d10d659ee6dd621dbaf76e20290f59ad4f35af/merged/mnt': Is a directory: OCI runtime error

I just removed all the references to /mnt and /run/media in toolbox and that temporarily resolved the issue for me.

[giuseppe]

@p1u3o just to be sure my fix addresses your issue, could you show me the output for stat /mnt?

[benhoman]

@giuseppe I'm having the same issue. The output of stat /mnt is:
File: /mnt -> var/mnt
Size: 7 Blocks: 0 IO Block: 4096 symbolic link
Device: fd01h/64769d Inode: 2621481 Links: 5
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:mnt_t:s0
Access: 2019-11-13 08:18:38.666995206 -0500
Modify: 2019-11-12 13:51:35.169000757 -0500
Change: 2019-11-13 06:09:45.432966095 -0500
Birth: 2019-11-12 13:51:35.169000757 -0500

[p1u3o]

If it's of any help I'm using XFS with SELinux set as permissive.

File: /mnt -> var/mnt
  Size: 7         	Blocks: 0          IO Block: 4096   symbolic link
Device: 822h/2082d	Inode: 134217875   Links: 4
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:mnt_t:s0
Access: 2019-11-13 09:13:30.879734789 +0000
Modify: 2019-11-11 09:36:01.148936755 +0000
Change: 2019-11-13 09:12:15.780417160 +0000
 Birth: 2019-11-11 09:36:01.148936755 +0000

[aaronuurman]

Same problem here.
After recreation of a container still can't enter.
File: /mnt -> var/mnt Size: 7 Blocks: 0 IO Block: 4096 symbolic link Device: fd00h/64768d Inode: 655408 Links: 4 Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:mnt_t:s0 Access: 2019-11-13 20:58:21.899394135 +0200 Modify: 2019-10-19 12:03:46.378316430 +0300 Change: 2019-11-13 19:37:41.427229814 +0200 Birth: 2019-10-19 12:03:46.378316430 +0300

[stephanmol]

I can confirm downgrading crun on SB31 using latest updates will result in working toolbox containers. Running "31.20191115.0 (2019-11-15T01:59:08Z)" and performed these steps:

1. Download crun-0.10.4-1.fc31 (https://kojipkgs.fedoraproject.org//packages/crun/0.10.4/1.fc31/x86_64/crun-0.10.4-1.fc31.x86_64.rpm)
2. rpm-ostree override replace ~/Downloads/crun-0.10.4-1.fc31.x86_64.rpm
3. systemctl reboot

Maybe this is helpful to someone getting no work done on SB as main desktop without having working toolbox containers.

I'm glad I found this post of @garrett which pointed me in this direction https://discussion.fedoraproject.org/t/toolbox-broken-again-crun-update-in-31-20191112-0/11369/8

[aaronuurman]

@stephanmol Thank you, got toolbox working again.

[aaronuurman]

A few days later, the toolbox again is broken.
Seems the problem is fixed in crun-0.10.6-1.fc3, according to this info containers/podman#4024.
To fix the issue perform similar steps as @stephanmol mentioned above:

• Download crun-0.10.6-1.fc3 (https://kojipkgs.fedoraproject.org//packages/crun/0.10.6/1.fc31/x86_64/crun-0.10.6-1.fc31.x86_64.rpm)

• rpm-ostree override replace ~/Downloads/crun-0.10.6-1.fc31.x86_64.rpm

• systemctl reboot

[garrett]

@aaronuurman crun-0.10.6-1.fc31.x86_64 is in Silverblue as of yesterday. You shouldn't need to override anymore.

I'm happily getting work done in containers on Silverblue right this very moment thanks to toolbox (and podman and crun). Hopefully it all works again for you too after an rpm-ostree update (and reboot)?

[stephanmol]

Should the override stay in place after running rpm-ostree upgrade I had to perform manually rpm-ostree override reset crun which made it used the latest version of crun which fixes the issue (yay!)

[debarshiray]

Thanks for sorting this out, @giuseppe !

[aaronuurman]

It seems override stays in place after rpm-ostree upgrade, probably that's why I did a new override.
Thank you for pointing it out 👍

[garrett]

FWIW, I did an rpm-ostree override reset -a and rebooted just to be sure I didn't have any overrides. (Specifying just crun or crun-0.10.6 didn't work. But resetting all did.)

The crun override wasn't showing, but I wanted to make sure it wasn't transparently there because it was the same version as what now ships in Silverblue. (I don't want to have some surprise in the future. 😉)

[debarshiray]

Let me re-purpose this issue to improve our handling of /mnt and /media to make things a bit more robust in the immediate short-term. We already do the same for /home.