Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

State database doesn't save storage options #2217

Closed
Silvanoc opened this issue Jan 24, 2019 · 15 comments

Comments

Projects
None yet
5 participants
@Silvanoc
Copy link

commented Jan 24, 2019

/kind bug

Description

State DB (Bolt) doesn't save storage options, therefore they are honored on the first run, but not on following runs.

Steps to reproduce the issue:

  1. Configure a storage option (e.g. ostree_repo = "/var/lib/ostree").

  2. Pull an image with debug level "looking" for the options output: sudo podman --log-level debug pull alpine:3.8 2>&1 | grep "overlay: "

  3. Pull another image with debug level "looking" for the options output: sudo podman --log-level debug pull alpine:3.7 2>&1 | grep "overlay: "

Describe the results you received:

On first pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

On second pull:

<nothing at all>

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

Describe the results you expected:

On first pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

On second pull:

time="..." level=debug msg="overlay: ostree_repo=/var/lib/ostree" 
time="..." level=debug msg="overlay: override_kernelcheck=true" 
time="..." level=warning msg="Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument" 

References available in OStree repository (ostree refs --repo=/var/lib/ostree):

containers-storage/d6da3c54c8f3a43215d51cc0d43eff9759624996e40aaf11e4be5a555c038c32
containers-storage/7bff100f35cb359a368537bb07829b055fe8e0b1cb01085a3a628ae9c187c7b8

Additional information you deem important (e.g. issue happens only occasionally):

If the state DB (bolt_state.db) is removed after the first run, then the output is the expected one.

Output of podman version:

Version:       1.0.0
Go Version:    go1.11.4
Git Commit:    ""
Built:         Mon Jan 14 23:12:09 2019
OS/Arch:       linux/amd64

Output of podman info:

host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: Unknown
    path: /usr/libexec/crio/conmon
    version: 'conmon version 1.13.0, commit: e8a2525c2e7f5ab057d5a2b5f1950be5643d8053-dirty'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 245493760
  MemTotal: 3923992576
  OCIRuntime:
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc5+dev
      commit: 079817cc26ec5292ac375bb9f47f373d33574949
      spec: 1.0.1-dev
  SwapFree: 8188518400
  SwapTotal: 8204779520
  arch: amd64
  cpus: 4
  hostname: ...
  kernel: 4.20.3-arch1-1-ARCH
  os: linux
  rootless: false
  uptime: 2h 24m 40.3s (Approximately 0.08 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage

Whereas the value of GraphOptions after first run is

  - overlay.ostree_repo=/var/lib/ostree
  - overlay.mountopt=nodev
  - overlay.override_kernel_check=true

Additional environment details (AWS, VirtualBox, physical, etc.):

Bare metal installation of Arch Linux. Libpod from AUR package: https://aur.archlinux.org/packages/libpod/

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 25, 2019

We store nothing related to images in the BoltDB state (except for references to which image our containers use). We also don't store storage configuration there, except to use for checking the storage configuration passed in by the user. I wonder if it isn't the storage configuration checks that are blowing up here, though. It could be overwriting some information used by the ostree backend.

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Jan 25, 2019

@mheon I don't think that it's only the OStree configuration being affected, since another option active override_kernelcheck=true is also getting missed.

If you are not saving any of those values, then it's probably the configuration check blowing up.

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Jan 25, 2019

@mheon I don't know if the presence of the BoltDB is changing the checks in any ways. Remember that removing the DB after each run helps podman honoring those options.

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 25, 2019

It seems like we're somehow blanking supplemental storage options when the DB's saved storage configuration is not empty

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 25, 2019

(This is strange, because I know we're not doing that for rootless Podman - the fuse-overlayfs mount program is preserved there)

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Jan 25, 2019

Due to an issue that I initially faced trying to use OStree with rootless containers I started playing around with these combinations as root. I could isolate the issue as root, but I would expect the same issue to appear rootless.

In fact it would probably explain by I'm having issues to get rootless containers using fuse-overlayfs although I installed it and configured mount_program as documented.

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 25, 2019

Hm.
@Silvanoc Can you pastebin your /etc/containers/storage.conf? I'm wondering if something might be affecting how we parse it.

(We have issues where we won't error on trying to load a storage config that is completely unreadable to us, but that doesn't explain why it works the first time and not the second)

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Jan 26, 2019

Let me pastebin everything.

/etc/containers/storage.conf -> https://pastebin.com/apTiF1QX

Debug output first run -> https://pastebin.com/1YUTuC4U
Debug output second run -> https://pastebin.com/8kDS6eC4

Debug output second run removing BoltDB -> https://pastebin.com/6nET34WK

BTW, the result of the second run is exactly the same if I wipe the whole /var/lib/containers and run everything from scratch removing BoltDB between the first and the second run.

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Jan 26, 2019

Wow! This is strange! Running the same commands rootless (pretty much same configuration, except for user paths and mount_program = "/usr/bin/fuse-overlayfs") works as expected! 😮

So the issue appears in root execution, but not in rootless execution. This might help you isolate the issue.

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Jan 27, 2019

I think it has to be https://github.com/containers/libpod/blob/master/pkg/util/utils.go#L322

For rootless cases, we force a reload of the storage configuration file, which pre-populates our storage config from storage.conf - including storage options

For non-rootless cases, once the DB has been created, we pre-populate a good deal of storage.conf from it (to try and ensure path consistency - if you change a directory after containers have been created, strange and undesirable things may happen). That seems to be overriding our normal check into storage.conf which would pick up the additional storage options.

I think we need to make that storage.conf reload unconditional, not for only rootless cases

@rhatdan

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

@giuseppe and I were playing in this code a while back. WDYT @giuseppe

@giuseppe

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

I agree with @mheon, storage.conf should probably be reloaded for root containers as well, so that if Podman honors any change happening there

@mheon

This comment has been minimized.

Copy link
Collaborator

commented Feb 5, 2019

Taking a look at this tomorrow

@mheon mheon self-assigned this Feb 5, 2019

mheon added a commit to mheon/libpod that referenced this issue Feb 5, 2019

Unconditionally refresh storage options from config
Due to our unconditionally setting some storage options, we
are not always reading storage options from storage.conf. This
can lead to some fields in the storage config (most notably extra
storage options) being ignored, despite being set in
storage.conf.

Resolve this by unconditionally refreshing our storage config
from storage.conf (this was previously only done for rootless
Podman)

Fixes containers#2217

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
@mheon

This comment has been minimized.

Copy link
Collaborator

commented Feb 5, 2019

#2268 should fix

@Silvanoc Any chance you can build a patched Podman and verify that the fix works?

@Silvanoc

This comment has been minimized.

Copy link
Author

commented Feb 5, 2019

@mheon sure! I'll provide any feedback directly on the PR.

mheon added a commit to mheon/libpod that referenced this issue Feb 6, 2019

Unconditionally refresh storage options from config
Due to our unconditionally setting some storage options, we
are not always reading storage options from storage.conf. This
can lead to some fields in the storage config (most notably extra
storage options) being ignored, despite being set in
storage.conf.

Resolve this by unconditionally refreshing our storage config
from storage.conf (this was previously only done for rootless
Podman)

Fixes containers#2217

Signed-off-by: Matthew Heon <matthew.heon@pm.me>

mheon added a commit to mheon/libpod that referenced this issue Feb 6, 2019

Unconditionally refresh storage options from config
Due to our unconditionally setting some storage options, we
are not always reading storage options from storage.conf. This
can lead to some fields in the storage config (most notably extra
storage options) being ignored, despite being set in
storage.conf.

Resolve this by unconditionally refreshing our storage config
from storage.conf (this was previously only done for rootless
Podman)

Fixes containers#2217

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.