New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

libpod: database configuration mismatch #1853

Closed
watersb opened this Issue Nov 22, 2018 · 9 comments

Comments

Projects
None yet
3 participants
@watersb

watersb commented Nov 22, 2018

[//]: kind bug

Description

After libpod upgrade, I can no longer access my containers that I had created with rootless podman.

Steps to reproduce the issue:

$ podman ps
error creating libpod runtime: database run root /run/user/1000/run does not match our run root /run/user/1000: database configuration mismatch

Describe the results you received:

Describe the results you expected:

Additional information

On 6 November 2018, I created some containers using podman via fedora-toolbox, which is a shell script that sets up sub-environments for mutable state such as a desktop software-developer environment.

https://github.com/debarshiray/fedora-toolbox.git

Output of podman version:

Version:       0.11.2-dev
Go Version:    go1.11.2
OS/Arch:       linux/amd64

Output of podman info:

$ podman info
could not get runtime: database run root /run/user/1000/run does not match our run root /run/user/1000: database configuration mismatch

Okay, so moving my old ${HOME}/.local/share/containers directory out of the way, and creating a new container, works fine:

$ cd .local/share/
$ mv containers ~/Desktop/old-containers
$ mv ~/Desktop/new-containers containers
$ podman info
host:
  BuildahVersion: 1.5-dev
  Conmon:
    package: podman-0.11.2-4.dev.gitea928f2.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: bef4d3d95e0dbc97e8da5e1c1215d965d5d375c2'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 13379923968
  MemTotal: 16621510656
  OCIRuntime:
    package: runc-1.0.0-57.dev.git9e5aa74.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc5+dev
      commit: beb7491d9e59858033ae56537ea2df60dc63a5a9
      spec: 1.0.1-dev
  SwapFree: 8380215296
  SwapTotal: 8380215296
  arch: amd64
  cpus: 4
  hostname: x230
  kernel: 4.20.0-0.rc3.git0.1.fc30.x86_64
  os: linux
  rootless: true
  uptime: 20m 50.43s
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/bwaters/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000

Additional environment details

I am running Fedora Silverblue Rawhide, which is quite new to me; I have used Gentoo/Funtoo Linux for 15 years. I have little experience with OCI but trying to learn.

$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181120.n.0 (2018-11-21T01:28:41Z)
                BaseCommit: ebb71a87dedfa3b765030cc202aa669f164619d5cba5934a216f19bb71addded
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: emacs fprintd-pam gnome-font-viewer opensc pam_yubico
                            powerline-fonts sway

  ostree://fedora-workstation:fedora/rawhide/x86_64/silverblue
                   Version: Rawhide.20181119.n.0 (2018-11-19T21:11:30Z)
                BaseCommit: 44fa5853d00e57cfd2592fa9e321de6c28257d2cce49f26f617ff0d569c79d1a
              GPGSignature: Valid signature by F1D8EC98F241AAF20DF69420EF3C111FCFC659B9
           LayeredPackages: emacs fprintd-pam gnome-font-viewer opensc pam_yubico
                            powerline-fonts sway
@mheon

This comment has been minimized.

Collaborator

mheon commented Nov 22, 2018

We are aware of this breaking change and are working on a fix - hopefully we'll get it in and get a new release cut sometime next week.

In the meantime, you can either use config files to override the directories in question and force the original paths (there's a brief guide to this in the TROUBLESHOOTING.md file in this repo) or downgrade to a version lower than 0.11.1

@watersb

This comment has been minimized.

watersb commented Nov 22, 2018

Thanks for the work! libpod is a good direction, make things simpler, hopefully.

Just wish to follow up, I have tried to implement the workaround with config files, and that just switches me between two error states. I am sorry that I am not more familiar as yet with the pieces at play here,



[libpod.conf.txt](https://github.com/containers/libpod/files/2606708/libpod.conf.txt)
$ podman ps
error creating libpod runtime: database tmp dir /run/user/1000/libpod/tmp does not match our tmp dir /run/user/1000/run: database configuration mismatch

$ podman ps
error creating libpod runtime: database run root /run/user/1000/run does not match our run root /run/user/1000: database configuration mismatch
@watersb

This comment has been minimized.

watersb commented Nov 22, 2018

${HOME}/.config/containers/libpod.conf

# libpod.conf is the default configuration file for all tools using libpod to
# manage containers

# Default transport method for pulling and pushing for images
image_default_transport = "docker://"

# Paths to look for a valid OCI runtime (runc, runv, etc)
runtime_path = [
	     "/usr/bin/runc",
	     "/usr/sbin/runc",
	     "/usr/local/bin/runc",
	     "/usr/local/sbin/runc",
	     "/sbin/runc",
	     "/bin/runc",
	     "/usr/lib/cri-o-runc/sbin/runc"
]

# Paths to look for the Conmon container manager binary
conmon_path = [
	    "/usr/libexec/podman/conmon",
	    "/usr/libexec/crio/conmon",
	    "/usr/local/lib/podman/conmon",
	    "/usr/local/libexec/crio/conmon",
	    "/usr/bin/conmon",
	    "/usr/sbin/conmon",
	    "/usr/lib/podman/bin/conmon",
	    "/usr/lib/crio/bin/conmon"
]

# Environment variables to pass into conmon
conmon_env_vars = [
		"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]

# CGroup Manager - valid values are "systemd" and "cgroupfs"
# cgroup_manager = "systemd"

# Directory for persistent libpod files (database, etc)
# By default, this will be configured relative to where containers/storage
# stores containers
# Uncomment to change location from this default
#static_dir = "/var/lib/containers/storage/libpod"

# Directory for temporary files. Must be tmpfs (wiped after reboot)
tmp_dir = "/run/user/1000/libpod/tmp"

# Maximum size of log files (in bytes)
# -1 is unlimited
max_log_size = -1

# Whether to use chroot instead of pivot_root in the runtime
no_pivot_root = false

# Directory containing CNI plugin configuration files
cni_config_dir = "/etc/cni/net.d/"

# Directories where the CNI plugin binaries may be located
cni_plugin_dir = [
	       "/usr/libexec/cni",
	       "/usr/lib/cni",
	       "/usr/local/lib/cni",
	       "/opt/cni/bin"
]

# Default CNI network for libpod.
# If multiple CNI network configs are present, libpod will use the network with
# the name given here for containers unless explicitly overridden.
# The default here is set to the name we set in the
# 87-podman-bridge.conflist included in the repository.
# Not setting this, or setting it to the empty string, will use normal CNI
# precedence rules for selecting between multiple networks.
cni_default_network = "podman"

# Default libpod namespace
# If libpod is joined to a namespace, it will see only containers and pods
# that were created in the same namespace, and will create new containers and
# pods in that namespace.
# The default namespace is "", which corresponds to no namespace. When no
# namespace is set, all containers and pods are visible.
#namespace = ""

# Default pause image name for pod pause containers
pause_image = "k8s.gcr.io/pause:3.1"

# Default command to run the pause container
pause_command = "/pause"

# Determines whether libpod will reserve ports on the host when they are
# forwarded to containers. When enabled, when ports are forwarded to containers,
# they are held open by conmon as long as the container is running, ensuring that
# they cannot be reused by other programs on the host. However, this can cause
# significant memory usage if a container has many ports forwarded to it.
# Disabling this can save memory.
#enable_port_reservation = true

# Default libpod support for container labeling
# label=true
@mheon

This comment has been minimized.

Collaborator

mheon commented Nov 23, 2018

Alright, it looks like you'd need a modified storage.conf as well to match the storage directories as well.
Follow roughly the same instructions as libpod.conf, except the file to copy is /etc/containers/storage.conf (same destination, ~/.config/containers), and don't comment anything out. Set driver to "vfs" (defaults to "overlay") and set runroot to "/run/user/1000/run" (from your error message). Set graphroot to "/home/$USERNAME/.local/share/containers/storage".

@mheon

This comment has been minimized.

Collaborator

mheon commented Nov 23, 2018

(Replace $USERNAME in graphroot with your own username)

@mheon

This comment has been minimized.

Collaborator

mheon commented Nov 23, 2018

(We're patching libpod to autogenerate these configs, so at least in the future we'll not need most of this setup if such things need to be adjusted again)

@giuseppe

This comment has been minimized.

Member

giuseppe commented Dec 3, 2018

Alternatively, if it is fine for you to destroy the storage you can more easily do it with: sudo rm -rf ~/.local/share/containers/storage/

@mheon

This comment has been minimized.

Collaborator

mheon commented Dec 3, 2018

#1918 should be a more permanent fix

@mheon

This comment has been minimized.

Collaborator

mheon commented Dec 5, 2018

With #1918 landed, this should be fixed. Release with the fix is coming soon.

@mheon mheon closed this Dec 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment