Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rootless: readonly volume mount fails with unhelpful diagnostic #2312

Closed
edsantiago opened this issue Feb 11, 2019 · 15 comments

Comments

Projects
None yet
4 participants
@edsantiago
Copy link
Collaborator

commented Feb 11, 2019

$ podman run --rm -v /tmp:/tmp:ro alpine date
container create failed: container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"rootfs_linux.go:58: mounting \\\"/tmp\\\" to rootfs \\\"/home/esm/.local/share/containers/storage/vfs/dir/a1450cd2ae39b3a8f6e8263064ca2fada1406395912bd4bdc8bfd4c171de6858\\\" at \\\"/home/esm/.local/share/containers/storage/vfs/dir/a1450cd2ae39b3a8f6e8263064ca2fada1406395912bd4bdc8bfd4c171de6858/tmp\\\" caused \\\"operation not permitted\\\"\""
: internal libpod error

If run with -it, the error message is even less helpful:

$ podman run -it --rm -v /tmp:/tmp:ro alpine date
error reading container (probably exited) json message: EOF

There is no AVC in either case. Running with --log-level=debug produces no useful output (i.e. nothing different from a successful run). Removing :ro yields success.

podman-1.0.0-1.git82e8011.fc29 and runc-1.0.0-67.dev.git12f6a99.fc29 ; kernel 4.19.8-300.fc29.x86_64

Unfortunately I can't reproduce on a fresh virt.

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Feb 11, 2019

I think we may have fixed this already on master

@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 11, 2019

I just built from 112a5ab and am still seeing the same problem.

edsantiago added a commit to edsantiago/libpod that referenced this issue Feb 13, 2019

get_ci_vm : allow running without sudo
More complicated than one would think. The first problem is that,
on certain (but not all) Fedora systems, podman cannot mount
volumes read-only (issue containers#2312). This is baffling, and since
it's not easily reproducible it's likely that the dev team
will not spend much effort on it. Workaround: instead of bind-
mounting /tmp read-only, bind-mount a *tempdir* (subdirectory)
read-write. This is actually cleaner in some ways but it
leads to complications with the paths we use and with cleanup.

Next, allow overriding the default image and allow asking
for no sudo:

    export GCLOUD_IMAGE=quay.io/edsantiago/gcloud_centos:latest
    export GCLOUD_SUDO=

(yes, that's an equal-sign and EOL. Just an empty string).

The third part, unfortunately, requires a custom image because
the as_dollar_user.sh script (the one that runs gcloud in a
container) is hardwired in a cevich image and needs tweaks
in order to detect rootless and avoid sudo.

Signed-off-by: Ed Santiago <santiago@redhat.com>
@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

I am taking a look

@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

it looks like a bug in runc.

To verify it, I've tried another runtime:

$ podman --runtime /usr/local/bin/crun run -ti --rm -v /tmp:/tmp:ro alpine sh -c 'date; echo hello > /tmp/foo'
Wed Feb 13 16:19:20 UTC 2019
sh: can't create /tmp/foo: Is a directory
@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

I'm really confused now because I have two systems, one that reproduces the failure and one that works cleanly, and both have runc-1.0.0-67.dev.git12f6a99.fc29.x86_64

@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

is there any difference between the two systems? Same kernel?

@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

Lots of differences in installed packages; I've been trying to reduce that, but no luck.

Kernels that fail: 4.19.8-300.fc29, 4.20.3-200
Kernels that pass: 4.18.16-300, 4.20.6-200

@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

it fails on 4.20.6-200.fc29.x86_64 and runc-1.0.0-67.dev.git12f6a99.fc29.x86_64 as well

@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

it fails on 4.20.6-200.fc29.x86_64

Not for me. That seems to eliminate kernel as the source of the problem. (I just tested on 4.20.7-200.fc29; pass)

@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 13, 2019

@giuseppe what's your ~/.config/containers/storage.conf ? Mine is:

[storage]
  driver = "overlay"
  runroot = "/run/user/14904"
  graphroot = "/home/esm/.local/share/containers/storage"
  [storage.options]
    mount_program = "/usr/bin/fuse-overlayfs"

I've also tried driver = "vfs" with mount_program = "". No difference. (I've also tried on LVM $HOME vs non-LVM; no difference)

@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 13, 2019

@edsantiago it is exactly the same, just different UID and home directory.

@rhatdan

This comment has been minimized.

Copy link
Member

commented Feb 23, 2019

@edsantiago @giuseppe Is this resolved?

@edsantiago

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 23, 2019

I'm still seeing it on master - 0969d72

@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 25, 2019

I think it depends from runc, but I'll need to investigate it further

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 25, 2019

podman: fix ro bind mounts if no* opts are on the source
This is a workaround for the runc issue:

opencontainers/runc#1247

If the source of a bind mount has any of nosuid, noexec or nodev, be
sure to propagate them to the bind mount so that when runc tries to
remount using MS_RDONLY, these options are also used.

Closes: containers#2312

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
@giuseppe

This comment has been minimized.

Copy link
Member

commented Feb 25, 2019

PR here: #2432

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 25, 2019

podman: fix ro bind mounts if no* opts are on the source
This is a workaround for the runc issue:

opencontainers/runc#1247

If the source of a bind mount has any of nosuid, noexec or nodev, be
sure to propagate them to the bind mount so that when runc tries to
remount using MS_RDONLY, these options are also used.

Closes: containers#2312

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 25, 2019

podman: fix ro bind mounts if no* opts are on the source
This is a workaround for the runc issue:

opencontainers/runc#1247

If the source of a bind mount has any of nosuid, noexec or nodev, be
sure to propagate them to the bind mount so that when runc tries to
remount using MS_RDONLY, these options are also used.

Closes: containers#2312

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe added a commit to giuseppe/libpod that referenced this issue Feb 25, 2019

podman: fix ro bind mounts if no* opts are on the source
This is a workaround for the runc issue:

opencontainers/runc#1247

If the source of a bind mount has any of nosuid, noexec or nodev, be
sure to propagate them to the bind mount so that when runc tries to
remount using MS_RDONLY, these options are also used.

Closes: containers#2312

Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.