ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay"

[ansemjo]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug

Description
Since the upgrade to podman 1.0.0 I get the above error in all sorts of situations during rootless operation. Mind you: I don't use the overlay driver at all. I did use it before though and I have fuse-overlayfs installed. This is basically a complete blocker for podman usage as soon as I have anything in storage, i.e. downloaded a single image or try to run a container.

Steps to reproduce the issue:

1. I'm not sure? Run rootless podman 1.0.0 on a btrfs filesystem?

2. Run any podman command

Describe the results you received:

Almost every command fails with the above error from the title. See for example this series of commands attempting to switch to the btrfs storage driver:

⦁ ansemjo @thinkmett ~ $ vi .config/containers/storage.conf 

⦁ ansemjo @thinkmett ~ $ cat .config/containers/storage.conf 
RunRoot = "/run/user/1000"
GraphRoot = "/home/ansemjo/.local/share/containers/storage"
GraphDriverName = "btrfs"

⦁ ansemjo @thinkmett ~ $ rm -rf .local/share/containers/storage/

⦁ ansemjo @thinkmett ~ $ podman pull alpine
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
Trying to pull docker.io/alpine:latest...Getting image source signatures
Copying blob cd784148e348: 2.10 MiB / 2.10 MiB [============================] 1s
Copying config 3f53bb00af94: 1.48 KiB / 1.48 KiB [==========================] 0s
Writing manifest to image destination
Storing signatures
3f53bb00af943dfdf815650be70c0fa7b426e56a66f5e3362b47a129d57d5991

⦁ ansemjo @thinkmett ~ $ podman pull alpine
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

125 ansemjo @thinkmett ~ $ ls -la .local/share/containers/storage/
total 4
drwx------ 1 ansemjo users 144 Jan 17 07:04 .
drwx------ 1 ansemjo users  14 Jan 17 07:04 ..
drwx------ 1 ansemjo users  20 Jan 17 07:04 btrfs
drwx------ 1 ansemjo users  30 Jan 17 07:04 btrfs-containers
drwx------ 1 ansemjo users 172 Jan 17 07:04 btrfs-images
drwx------ 1 ansemjo users 198 Jan 17 07:04 btrfs-layers
drwx------ 1 ansemjo users  26 Jan 17 07:04 libpod
drwx------ 1 ansemjo users   0 Jan 17 07:04 mounts
drwx------ 1 ansemjo users   0 Jan 17 07:04 tmp
-rw------- 1 ansemjo users  64 Jan 17 07:04 storage.lock

⦁ ansemjo @thinkmett ~ $ podman run --rm -it alpine
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

125 ansemjo @thinkmett ~ $ podman info
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Describe the results you expected:

I don't expect to see any errors related to overlay fs when it's not configured to be used?

Additional information you deem important (e.g. issue happens only occasionally):

• Deleting the storage location enables me to use podman info once but as soon as I run any command that seems to populate the storage the errors reappears. That also means a second podman info will fail.

• I've had a little different results yesterday, I could even start a container and occasionally run commands even though they printed the same error at the beginning.

• The filesystem is mounted with user_subvol_rm_allowed.

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    "5238ca20b3c7ee7b67619f159bcc89ecee095f99"
Built:         Sat Jan 12 14:57:45 2019
OS/Arch:       linux/amd64

Output of podman info:

$ podman info
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: Unknown
    path: /usr/libexec/crio/conmon
    version: 'conmon version 1.13.0, commit: e8a2525c2e7f5ab057d5a2b5f1950be5643d8053-dirty'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 3808534528
  MemTotal: 8259203072
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc5+dev
      commit: 079817cc26ec5292ac375bb9f47f373d33574949
      spec: 1.0.1-dev
  SwapFree: 4294963200
  SwapTotal: 4294963200
  arch: amd64
  cpus: 4
  hostname: thinkmett
  kernel: 4.20.1-arch1-1-ARCH
  os: linux
  rootless: true
  uptime: 21m 56.71s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/ansemjo/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: btrfs
  GraphOptions: null
  GraphRoot: /home/ansemjo/.local/share/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v4.19.1 '
    Library Version: "102"
  ImageStore:
    number: 0
  RunRoot: /run/user/1000

$ podman info
ERRO[0000] 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay" 
could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over btrfs at "/home/ansemjo/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

Additional environment details (AWS, VirtualBox, physical, etc.):

Physical.

[ansemjo]

While I was still writing the original issue, I checked if I had overlay configured in /etc/containers/storage.conf and indeed I had. I changed that to btrfs as well, deleted my local storage again and rerun a couple of commands. So far everything works as expected.

The issue sort-of remains: why does the /etc/containers/storage.conf take any precedence over my .config/containers/storage.conf during rootless operation? And why does it complain about overlay when it apparently does not create any overlays in my storage location (see ls -la .. output above)?

[debarshiray]

Based on my very limited understanding, I think this camel case format for storage.conf is wrong:

⦁ ansemjo @thinkmett ~ $ cat .config/containers/storage.conf 
RunRoot = "/run/user/1000"
GraphRoot = "/home/ansemjo/.local/share/containers/storage"
GraphDriverName = "btrfs"

The correct format, I am told, uses lower case strings and has a slightly different layout. See #2158

I have found that podman randomly picks one of the two formats when initializing a new storage.conf, which is how I think many have ended up with the wrong format. See #2158

[ansemjo]

Right .. I switched my config to the lowercase format and so far can't reproduce my issue, even when changing the system /etc/containers/storage.conf back to overlay.

$ cat ~/.config/containers/storage.conf 
[storage]

# Default Storage Driver
driver = "btrfs"

# Temporary storage location
runroot = "/run/user/1000"

# Primary Read/Write location of container storage
graphroot = "/home/ansemjo/.local/share/containers/storage"

[giuseppe]

yes it was a bug in an older version of Podman that was creating the wrong config file, so basically the config file wasn't honored at all and the default configuration was used (by default we use overlay when fuse-overlayfs is installed, otherwise fallback to vfs).

I am closing this bug for now, as the error was in the configuration created by an old version of Podman. But please feel free to comment more if you have more questions/issues

[ansemjo]

Yes, closing is fine by me, thanks.

On a more general note: I found it pretty hard to find any information on configuring storage drivers. I.e. what drivers are available? What conditions must hold? What options can be used? etc.
I only stumbled upon the fact that there is a btrfs driver because I saw it being disabled in some test configurations in this repository when searching for GraphDriver.

I understand that it is not entirely fair to compare podman to Docker in terms of documentation yet. But I like their documentation a lot.

[giuseppe]

On a more general note: I found it pretty hard to find any information on configuring storage drivers. I.e. what drivers are available? What conditions must hold? What options can be used? etc.
I only stumbled upon the fact that there is a btrfs driver because I saw it being disabled in some test configurations in this repository when searching for GraphDriver.

I assumed that information was in man storage.conf, we should probably add it there.