[podman] kernel does not support overlay fs #2659

tobwen · 2019-03-14T21:04:47Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
podman complains that my kernel doesn't support overlayfs. But it does (the kernel does and all the tools are installed and in search-path).

Steps to reproduce the issue:

~/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay run busybox sh

Describe the results you received:


INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Using slirp4netns netmode
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: operation not permitted
DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: operation not permitted
ERRO[0000] 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay"
ERRO[0000] error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Additional information you deem important (e.g. issue happens only occasionally):

fuse3 and fuse-overlayfs are installed and working. I'm using ext4 only. Might be connected with #545

Output of podman version:

INFO[0000] running as rootless
Version:            1.2.0-dev
RemoteAPI Version:  1
Go Version:         go1.11.5
Git Commit:         c7aa191c8bcb83e8f5064d458d6604fcdf0e41d9
Built:              Thu Mar 14 19:52:43 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

~/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc info --debug
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver vfs
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "vfs"
debug:
  compiler: gc
  git commit: c7aa191c8bcb83e8f5064d458d6604fcdf0e41d9
  go version: go1.11.5
  podman version: 1.2.0-dev
host:
  BuildahVersion: 1.8-dev
  Conmon:
    package: Unknown
    path: /home/podman/podman/usr/local/libexec/crio/conmon
    version: 'conmon version , commit: cc2b49590a485da9bd358440f92f219dfd6b230f'
  Distribution:
    distribution: debian
    version: "9"
  MemFree: 34703171584
  MemTotal: 38205562880
  OCIRuntime:
    package: Unknown
    path: /home/podman/podman/usr/local/sbin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
      spec: 1.0.1-dev
  SwapFree: 8586784768
  SwapTotal: 8586784768
  arch: amd64
  cpus: 8
  hostname: pgsql
  kernel: 4.19.0-0.bpo.2-amd64
  os: linux
  rootless: true
  uptime: 103h 35m 21.42s (Approximately 4.29 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - quay.io
store:
  ConfigFile: /home/podman/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/podman/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 0
  RunRoot: /home/podman/rundir
  VolumePath: /home/podman/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Debian 9 on VirtualBox 6 on Windows 10 on AMD 64

The text was updated successfully, but these errors were encountered:

rhatdan · 2019-03-15T13:36:05Z

Does this work if you run it as root?

If yes, then cleanup your homedir, and try again.

rm -rf ~/.config/container ~/.local/share/containers

tobwen · 2019-03-15T13:58:46Z

Does this work if you run it as root?

Nope, sorry.

# fuse-overlayfs
crun: upperdir not specified
# rm -rf /tmp/user /tmp/libpod-rundir* /home/podman/.config/container 
# rm -rf ~/.config/container ~/.local/share/containers
# /home/podman/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay run busybox sh
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /var/run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /var/run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
ERRO[0000] error creating libpod runtime: database storage graph driver  does not match our storage graph driver overlay: database configuration mismatch

If yes, then cleanup your homedir, and try again.

Of course, I've cleaned everything up before (just forgot to mention)

mheon · 2019-03-15T14:01:10Z

ERRO[0000] error creating libpod runtime: database storage graph driver  does not match our storage graph driver overlay: database configuration mismatch

Woah. That should not be happening, that - we should catch that and handle it natively...

mheon · 2019-03-15T14:16:05Z

#2668 to fix the validation issue there. Though I'm very curious as to how the database managed to get an empty string storage driver.

mheon · 2019-03-15T14:18:17Z

Theory on that: we reload storage.DefaultStoreOptions in NewRuntime() before DB validation runs, which leads us to cases like this when the file we reload from has fields left empty.

Should probably catch that earlier, with better error messages.

tobwen · 2019-03-15T15:02:58Z

I've build #2668 and tested on non-root and root. Tested with overlay, overlayfs and fuse-overlayfs and without --storage-driver. Only overlay worked.

/home/podman/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlayfs --storage-opt /home/podman/podman/usr/local/bin/fuse-overlayfs run busybox sh

Result (non-root):

$ /home/podman/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay run busybox sh
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Using slirp4netns netmode
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /home/podman/rundir
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /home/podman/rundir/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay test mount with multiple lowers failed mountfrom re-exec error: exit status 1: output: operation not permitted
DEBU[0000] overlay test mount with a single lower failed mountfrom re-exec error: exit status 1: output: operation not permitted
ERRO[0000] 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay"
ERRO[0000] error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

tobwen · 2019-03-15T15:09:40Z

Interesting, this command line works: #2662 (comment)

tobwen · 2019-03-16T18:24:05Z

Hmm, I'm still getting this problem without #2668

$ rm -rf /tmp/lib* /tmp/user* ~/.config/containers ~/.local/share/containers ~/rundir /tmp/run-*
$ /home/podman/podman/usr/local/bin/podman run --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay --storage-opt "overlay.mount_program=/home/podman/podman/usr/local/bin/fuse-overlayfs" run busybox sh
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1001
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-1001/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
ERRO[0000] error creating libpod runtime: Error running /home/podman/podman/usr/local/bin/podman info while refreshing state: time="2019-03-16T18:20:13Z" level=error msg="User-selected graph driver "vfs" overwritten by graph driver "overlay" from database - delete libpod local files to resolve"
time="2019-03-16T18:20:13Z" level=error msg="'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay""
Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

mheon · 2019-03-16T20:19:22Z

@giuseppe I think the core of our issue is rootless refresh. We're calling podman info but not passing in CLI arguments for changed settings. I can definitely see this breaking with podman --config, for example. Maybe it makes sense to dump the final configuration we used in the runtime to disk and the specify --config to info? It has the issue of not saving storage options, but we can manually set those with CLI flags.

(I also think the storage-opt thing probably also breaks podman cleanup if you manually specified fuse-overlayfs at the command line).

giuseppe · 2019-03-17T09:01:25Z

@giuseppe I think the core of our issue is rootless refresh. We're calling podman info but not passing in CLI arguments for changed settings. I can definitely see this breaking with podman --config, for example. Maybe it makes sense to dump the final configuration we used in the runtime to disk and the specify --config to info? It has the issue of not saving storage options, but we can manually set those with CLI flags.

this is what I've attempted to do with #2670 but apparently it is still not working as it should :/ I'll debug it further and check the case where fuse-overlayfs is not in the $PATH

tobwen · 2019-03-17T09:38:37Z

If fuse-overlayfs is in $PATH, everything works as expected. Without, it fails with the error reported.

giuseppe · 2019-03-17T09:46:51Z

@tobwen can you please check with the last patch I've added to the PR?

tobwen · 2019-03-17T10:59:30Z

Pulled, checked out, built, installed, ran:

$ git fetch origin pull/2670/head:pr2670c && git checkout pr2670c
$ make clean && make
$ make install install.config install.completions DESTDIR=/home/podman/podman
$ rm -rf /tmp/lib* /tmp/user* ~/rundir /tmp/run-* ~/.config/containers ~/.local/share/containers 

$ /home/podman/podman/usr/local/bin/podman run --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay --storage-opt "overlay.mount_program=/home/podman/podman/usr/local/bin/fuse-overlayfs" run busybox sh
INFO[0000] running as rootless
DEBU[0000] Initializing boltdb state at /home/podman/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/podman/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1001
DEBU[0000] Using static dir /home/podman/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-1001/libpod/tmp
DEBU[0000] Using volume path /home/podman/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
ERRO[0000] error creating libpod runtime: Error running /home/podman/podman/usr/local/bin/podman info while refreshing state: time="2019-03-17T10:58:19Z" level=error msg="'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay""
Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/home/podman/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver
: exit status 125

$ ls /home/podman/podman/usr/local/bin/fuse-overlayfs
/home/podman/podman/usr/local/bin/fuse-overlayfs

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

@giuseppe Sorry, doesn't seem to work.

giuseppe · 2019-03-17T11:51:51Z

how do the generated files under ~/.config/containers look like?

tobwen · 2019-03-17T11:55:23Z

$ ls ~/.config/containers
libpod.conf  storage.conf

$ cat libpod.conf
volume_path = "/home/podman/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime = "/home/podman/podman/usr/local/sbin/runc"
conmon_path = ["/home/podman/podman/usr/local/libexec/crio/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
init_path = "/usr/libexec/podman/catatonit"
static_dir = "/home/podman/.local/share/containers/storage/libpod"
tmp_dir = "/tmp/run-1001/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
network_cmd_path = "/home/podman/podman/usr/local/bin/slirp4netns"
num_locks = 2048
EventsLogFilePath = ""

[runtimes]
  runc = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]

$ cat storage.conf
[storage]
  driver = "overlay"
  runroot = "/tmp/run-1001"
  graphroot = "/home/podman/.local/share/containers/storage"
  [storage.options]
    mount_program = ""

giuseppe · 2019-03-17T12:10:42Z

--storage-driver and --storage-opt are global flags, not for run. Also, you have two run in your command.

Please try:

$ /home/podman/podman/usr/local/bin/podman --log-level=debug --conmon /home/podman/podman/usr/local/libexec/crio/conmon --network-cmd-path /home/podman/podman/usr/local/bin/slirp4netns --runtime /home/podman/podman/usr/local/sbin/runc --storage-driver overlay --storage-opt "overlay.mount_program=/home/podman/podman/usr/local/bin/fuse-overlayfs" run busybox sh

tobwen · 2019-03-17T12:17:15Z

Oh god, I'm so sorry, I didn't see the first run. I am so sorry. It works as expected now.

Maybe we should warn the user about two commands? I apologize very much for my boundless stupidity.

mheon · 2019-03-17T12:20:39Z

For some reason Cobra seems to allow mixing global and local flags, so commands like podman run --log-level=debug -t -i fedora bash now work, and actually enable debug logging successfully. I don't think I mind this - explaining to folks where global flags had to go was tedious.

mheon · 2019-03-17T12:35:18Z

@tobwen I don't think that's actually being handled as a duplicated command - when I tried locally, we interpreted the second run as the name of the image, and tried pulling images named run. Any arguments after that second run were going to be passed into the container when it was created, not Podman itself. Not much we can do about this if we want to duplicate the Docker CLI.

tobwen · 2019-03-17T13:14:31Z

@mheon Okay, I'll take care of this in future. btw, some applications split global and local flags by blah -- blah, but I understand the D-word compatibility.

Thanks for your good work. Would be happy to see it in master :-)

split the generation for the default storage.conf and when we write it if not existing for a rootless user. This is necessary because during the startup we might be overriding the default configuration through --storage-driver and --storage-opt, that would not be written down to the storage.conf file we generated. Closes: containers#2659 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

tobwen · 2019-10-19T02:12:52Z

The problem is back once again:

/home/tobwen/podman/usr/local/bin/podman --log-level=debug --tmpdir /tmp/user/1000/libpod/tmp --conmon /home/tobwen/podman/usr/local/bin/conmon --network-cmd-path /home/tobwen/podman/usr/local/bin/slirp4netns --runtime /home/tobwen/podman/usr/local/bin/runc --storage-driver overlay --storage-opt "overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs" run hello-world

podman seems to use fuse-overlayfs on the first run, but doesn't store the setting (cat ~/.config/containers/storage.conf)

[storage]
  driver = "vfs"
  runroot = "/tmp/run-1000"
  graphroot = "/home/tobwen/.local/share/containers/storage"
  [storage.options]
    size = ""
    remap-uids = ""
    remap-gids = ""
    ignore_chown_errors = ""
    remap-user = ""
    remap-group = ""
    mount_program = ""
    mountopt = ""
    [storage.options.thinpool]
      autoextend_percent = ""
      autoextend_threshold = ""
      basesize = ""
      blocksize = ""
      directlvm_device = ""
      directlvm_device_force = ""
      fs = ""
      log_level = ""
      min_free_space = ""
      mkfsarg = ""
      mountopt = ""
      use_deferred_deletion = ""
      use_deferred_removal = ""
      xfs_nospace_max_retries = ""

Here's the complete debug output:

DEBU[0000] using conmon: "/home/tobwen/podman/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/tobwen/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/tobwen/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1000
DEBU[0000] Using static dir /home/tobwen/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/tobwen/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] using conmon: "/home/tobwen/podman/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/tobwen/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/tobwen/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1000
DEBU[0000] Using static dir /home/tobwen/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/tobwen/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] Podman detected system restart - performing state refresh
INFO[0000] running as rootless
DEBU[0000] using conmon: "/home/tobwen/podman/usr/local/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/tobwen/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/tobwen/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1000
DEBU[0000] Using static dir /home/tobwen/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/tobwen/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] using runtime "/home/tobwen/podman/usr/local/bin/runc"
DEBU[0000] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]docker.io/library/hello-world:latest"
DEBU[0000] reference "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]docker.io/library/hello-world:latest" does not resolve to an image ID
DEBU[0000] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]localhost/hello-world:latest"
DEBU[0000] reference "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]localhost/hello-world:latest" does not resolve to an image ID
DEBU[0000] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]docker.io/library/hello-world:latest"
Trying to pull docker.io/library/hello-world...
DEBU[0000] reference rewritten from 'docker.io/library/hello-world:latest' to 'docker.io/library/hello-world:latest'
DEBU[0000] Trying to pull "docker.io/library/hello-world:latest"
DEBU[0000] Credentials not found
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  No signature storage configuration found for docker.io/library/hello-world:latest
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0005] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0005] GET https://auth.docker.io/token?scope=repository%3Alibrary%2Fhello-world%3Apull&service=registry.docker.io
DEBU[0010] GET https://registry-1.docker.io/v2/library/hello-world/manifests/latest
DEBU[0011] Using blob info cache at /home/tobwen/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0011] Source is a manifest list; copying (only) instance sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a
DEBU[0011] GET https://registry-1.docker.io/v2/library/hello-world/manifests/sha256:92c7f9c92844bbbb5d0a101b22f7c2a7949e40f8ea90c8b3bc396879d95e899a
DEBU[0011] IsRunningImageAllowed for image docker:docker.io/library/hello-world:latest
DEBU[0011]  Using default policy section
DEBU[0011]  Requirement 0: allowed
DEBU[0011] Overall: allowed
DEBU[0011] Downloading /v2/library/hello-world/blobs/sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e
DEBU[0011] GET https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e
Getting image source signatures
DEBU[0022] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v1+json]
DEBU[0022] ... will first try using the original manifest unmodified
DEBU[0022] [graphdriver] trying provided driver "overlay"
DEBU[0022] overlay: mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs
DEBU[0022] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0022] Downloading /v2/library/hello-world/blobs/sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced
DEBU[0022] GET https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:1b930d010525941c1d56ec53b97bd057a67ae1865eebf042686d2a2d18271ced
DEBU[0022] Detected compression format gzip
DEBU[0022] Using original blob without modification
Copying blob 1b930d010525 done
DEBU[0023] No compression detected
DEBU[0023] Using original blob without modification
Copying config fce289e99e done
Writing manifest to image destination
Storing signatures
DEBU[0023] Applying tar in /home/tobwen/.local/share/containers/storage/overlay/af0b15c8625bb1938f1d7b17081031f649fd14e6b233688eea3c5483994a66a3/diff
DEBU[0025] setting image creation date to 2019-01-01 01:29:27.650294696 +0000 UTC
DEBU[0025] created new image ID "fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] set names of image "fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e" to [docker.io/library/hello-world:latest]
DEBU[0025] saved image metadata "{}"
DEBU[0025] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]docker.io/library/hello-world:latest"
DEBU[0025] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]@fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] exporting opaque data as blob "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]@fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] exporting opaque data as blob "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]@fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] No hostname set; container's hostname will default to runtime default
DEBU[0025] Using slirp4netns netmode
DEBU[0025] created OCI spec and options for new container
DEBU[0025] Allocated lock 18 for container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b
DEBU[0025] parsed reference into "[overlay@/home/tobwen/.local/share/containers/storage+/tmp/run-1000:overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs]@fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] exporting opaque data as blob "sha256:fce289e99eb9bca977dae136fbe2a82b6b7d4c372474c9235adc1741675f587e"
DEBU[0025] created container "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b"
DEBU[0025] container "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b" has work directory "/home/tobwen/.local/share/containers/storage/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata"
DEBU[0025] container "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b" has run directory "/tmp/run-1000/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata"
DEBU[0025] New container created "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b"
DEBU[0025] container "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b" has CgroupParent "/libpod_parent/libpod-f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b"
DEBU[0025] Not attaching to stdin
DEBU[0025] Made network namespace at /tmp/run-1000/netns/cni-fc85f50e-dacd-c4a3-d770-318d89771ece for container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b
DEBU[0025] overlay: mount_data=lowerdir=/home/tobwen/.local/share/containers/storage/overlay/l/5Z6GYFJG5XDTRCNWFI4VQVEDGC,upperdir=/home/tobwen/.local/share/containers/storage/overlay/42b17383ed7c6c0fb9d1f02161b85744f76b000200a3383766f5535347478e6d/diff,workdir=/home/tobwen/.local/share/containers/storage/overlay/42b17383ed7c6c0fb9d1f02161b85744f76b000200a3383766f5535347478e6d/work
DEBU[0025] slirp4netns command: /home/tobwen/podman/usr/local/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox -c -e 3 -r 4 --netns-type=path /tmp/run-1000/netns/cni-fc85f50e-dacd-c4a3-d770-318d89771ece tap0
DEBU[0025] mounted container "f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b" at "/home/tobwen/.local/share/containers/storage/overlay/42b17383ed7c6c0fb9d1f02161b85744f76b000200a3383766f5535347478e6d/merged"
DEBU[0025] Created root filesystem for container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b at /home/tobwen/.local/share/containers/storage/overlay/42b17383ed7c6c0fb9d1f02161b85744f76b000200a3383766f5535347478e6d/merged
DEBU[0025] skipping loading default AppArmor profile (rootless mode)
DEBU[0025] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0025] Created OCI spec for container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b at /home/tobwen/.local/share/containers/storage/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata/config.json
DEBU[0025] /home/tobwen/podman/usr/local/bin/conmon messages will be logged to syslog
DEBU[0025] running conmon: /home/tobwen/podman/usr/local/bin/conmon  args="[--api-version 1 -c f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b -u f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b -r /home/tobwen/podman/usr/local/bin/runc -b /home/tobwen/.local/share/containers/storage/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata -p /tmp/run-1000/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata/pidfile -l k8s-file:/home/tobwen/.local/share/containers/storage/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata/ctr.log --exit-dir /tmp/user/1000/libpod/tmp/exits --socket-dir-path /tmp/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /tmp/run-1000/overlay-containers/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/userdata/conmon.pid --exit-command /home/tobwen/podman/usr/local/bin/podman --exit-command-arg --root --exit-command-arg /home/tobwen/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/run-1000 --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /tmp/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /home/tobwen/podman/usr/local/bin/runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/home/tobwen/podman/usr/local/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0025] Received: 6238
INFO[0025] Got Conmon PID as 6226
DEBU[0025] Created container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b in OCI runtime
DEBU[0025] Attaching to container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b
DEBU[0025] connecting to socket /tmp/user/1000/libpod/tmp/socket/f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b/attach
DEBU[0025] Starting container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b with command [/hello]
DEBU[0025] Started container f0590c6c3c996243a5ed61ee336d737e803f55fbc2500892a6eb1b15684cc84b

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

DEBU[0025] Enabling signal proxying

rhatdan · 2019-10-19T11:19:58Z

@giuseppe This looks like the storage.conf is not setting the override paths into the storage.conf file that it creates.

make sure the user overrides are stored in the configuration file when first created. Closes: containers#2659 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

giuseppe · 2019-10-21T10:07:56Z

PR here: #4309

tobwen · 2019-10-21T19:05:01Z

Tested, works.

[storage]
  driver = "overlay"
  runroot = "/tmp/run-1000"
  graphroot = "/home/tobwen/.local/share/containers/storage"
  [storage.options]
    mount_program = "/home/tobwen/podman/usr/local/bin/fuse-overlayfs"

make sure the user overrides are stored in the configuration file when first created. Closes: containers#2659 Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>

thiagolinhares · 2022-07-27T19:19:52Z

Tested, works.

[storage]
  driver = "overlay"
  runroot = "/tmp/run-1000"
  graphroot = "/home/tobwen/.local/share/containers/storage"
  [storage.options]
    mount_program = "/home/tobwen/podman/usr/local/bin/fuse-overlayfs"

Same here. Worked like a charm setting this to my storage.conf

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 14, 2019

tobwen changed the title ~~kernel does not support overlay fs~~ [podman] kernel does not support overlay fs Mar 14, 2019

rhatdan assigned giuseppe Mar 15, 2019

tobwen mentioned this issue Mar 16, 2019

[podman] path to fuse-overlayfs #2662

Closed

giuseppe mentioned this issue Mar 17, 2019

rootless: write the custom config file before reload #2670

Merged

openshift-merge-robot closed this as completed in 232b46a Mar 18, 2019

giuseppe reopened this Oct 21, 2019

giuseppe mentioned this issue Oct 21, 2019

rootless: write storage overrides to the conf file #4309

Merged

openshift-merge-robot closed this as completed in #4309 Oct 21, 2019

PedroRegisPOAR mentioned this issue Apr 18, 2021

Notes ES-Nix/podman-rootless#2

Open

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[podman] kernel does not support overlay fs #2659

[podman] kernel does not support overlay fs #2659

tobwen commented Mar 14, 2019 •

edited

Loading

rhatdan commented Mar 15, 2019

tobwen commented Mar 15, 2019

mheon commented Mar 15, 2019

mheon commented Mar 15, 2019

mheon commented Mar 15, 2019

tobwen commented Mar 15, 2019

tobwen commented Mar 15, 2019

tobwen commented Mar 16, 2019

mheon commented Mar 16, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019 •

edited

Loading

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

mheon commented Mar 17, 2019

mheon commented Mar 17, 2019

tobwen commented Mar 17, 2019

tobwen commented Oct 19, 2019

rhatdan commented Oct 19, 2019

giuseppe commented Oct 21, 2019

tobwen commented Oct 21, 2019

thiagolinhares commented Jul 27, 2022

[podman] kernel does not support overlay fs #2659

[podman] kernel does not support overlay fs #2659

Comments

tobwen commented Mar 14, 2019 • edited Loading

rhatdan commented Mar 15, 2019

tobwen commented Mar 15, 2019

mheon commented Mar 15, 2019

mheon commented Mar 15, 2019

mheon commented Mar 15, 2019

tobwen commented Mar 15, 2019

tobwen commented Mar 15, 2019

tobwen commented Mar 16, 2019

mheon commented Mar 16, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019 • edited Loading

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

giuseppe commented Mar 17, 2019

tobwen commented Mar 17, 2019

mheon commented Mar 17, 2019

mheon commented Mar 17, 2019

tobwen commented Mar 17, 2019

tobwen commented Oct 19, 2019

rhatdan commented Oct 19, 2019

giuseppe commented Oct 21, 2019

tobwen commented Oct 21, 2019

thiagolinhares commented Jul 27, 2022

tobwen commented Mar 14, 2019 •

edited

Loading

tobwen commented Mar 17, 2019 •

edited

Loading