Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: database libpod root directory (staticdir) does not match our libpod root directory (staticdir) #20872

Closed
kisame217 opened this issue Dec 1, 2023 · 24 comments · Fixed by #20874
Assignees
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. regression

Comments

@kisame217
Copy link

Issue Description

Starting with version 4.8.0, I get the error below.

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

I'm on Fedora Kinoite 39. No problems with version 4.7.2.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Upgraded to 4.8.0 from 4.7.2.
  2. Try to start a container
podman --log-level debug start postgres

Describe the results you received

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start postgres) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/<user>/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Overriding graph root "/var/home/<user>/.local/share/containers/storage" with "/home/<user>/.local/share/containers/storage" from database 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/<user>/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/<user>/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/<user>/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch
DEBU[0000] Shutting down engines 

Describe the results you expected

Container should start normally.

podman info output

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

OS: Fedora Kinoite 39
Arch: x86_64

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Running of bare metal.

Additional information

None.

@kisame217 kisame217 added the kind/bug Categorizes issue or PR as related to a bug. label Dec 1, 2023
@yann-soubeyrand
Copy link

Same on Fedora 39 Silverblue (I guess it’s expected since it affects Kinoite 39).

@Luap99
Copy link
Member

Luap99 commented Dec 1, 2023

What is the value of $HOME?

@Luap99
Copy link
Member

Luap99 commented Dec 1, 2023

Going through the git log #20088 looks like the only think which touched this area (although I cannot say for sure it caused this until I can reproduce).
cc @mheon @vrothberg

@yann-soubeyrand
Copy link

What is the value of $HOME?

/home/yann for me. It’s the correct path to my home directory. However, /home is a symlink to /var/home on Silverblue (and I guess on Kinoite too).

@Luap99
Copy link
Member

Luap99 commented Dec 1, 2023

Yeah I think the issue is because of this, previously we did not expand the symlink and now we do. Thus our db logic thinks you changed the config and errors out.

@Luap99
Copy link
Member

Luap99 commented Dec 1, 2023

Does it work if you set HOME=/var/home/yann?

@mheon
Copy link
Member

mheon commented Dec 1, 2023

We can probably fix this in the logic for checking the current config against the DB - eval symlinks on both paths (the one in the DB and the one we're trying to use) and compare the final paths. We'd need fixes for both Bolt and SQLite that way though

@Luap99
Copy link
Member

Luap99 commented Dec 1, 2023

reproducer on a not rpm-ostree system:

# create symlink for home
$ ln -s /home /tmp/home
# run system reset to start fresh
$ podman system reset

# run old podman 4.7.2, you need to change HOME var to your username of course
$ HOME=/tmp/home/pholzing podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
# run podman 4.8 or main
$ HOME=/tmp/home/pholzing bin/podman ps
Error: database libpod root directory (staticdir) "/tmp/home/pholzing/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/home/pholzing/.local/share/containers/storage/libpod": database configuration mismatch

@yann-soubeyrand
Copy link

Does it work if you set HOME=/var/home/yann?

Unfortunately I cannot test for now, I’ve rolled back my system (it’s my workstation). However, if I understand correctly, you’re now pretty sure this issue is due to symlink expansion and me confirming it doesn’t seem to be needed anymore.

mheon added a commit to mheon/libpod that referenced this issue Dec 1, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
@mheon
Copy link
Member

mheon commented Dec 1, 2023

#20874 to fix

mheon added a commit to mheon/libpod that referenced this issue Dec 1, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
mheon added a commit to mheon/libpod that referenced this issue Dec 1, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
mheon added a commit to mheon/libpod that referenced this issue Dec 1, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
mheon added a commit to mheon/libpod that referenced this issue Dec 1, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
@aanno
Copy link

aanno commented Dec 2, 2023

#20874 is still open. Is there a work-around? I tried to change HOME, but with no success.

@kisame217
Copy link
Author

#20874 is still open. Is there a work-around? I tried to change HOME, but with no success.

How did you change home? I set my home in /etc/passwd file. Also had to delete the containers folder in .local/share to reset the db config after I took backups of my container.

mheon added a commit to mheon/libpod that referenced this issue Dec 2, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
@mheon
Copy link
Member

mheon commented Dec 2, 2023

Setting static_dir in containers.conf is more likely to work. If your error message is:

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

Then set static_dir to the "database libpod root directory" (in this case /home/<user>/.local/share/containers/storage/libpod)

@elken
Copy link

elken commented Dec 2, 2023

Looks like I timed it well to run into this.

Setting static_dir in containers.conf is more likely to work. If your error message is:

Error: database libpod root directory (staticdir) "/home/<user>/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/<user>/.local/share/containers/storage/libpod": database configuration mismatch

Then set static_dir to the "database libpod root directory" (in this case /home/<user>/.local/share/containers/storage/libpod)

Have just tried with the following file to no luck
image

Have tried both combinations of home paths.

@mheon
Copy link
Member

mheon commented Dec 2, 2023 via email

@elken
Copy link

elken commented Dec 2, 2023

Looks pile it is now complaining about volume path, also in containers.conf (I think named volume_path but not sure?). Same procedure for the new error but different key in containers.conf basically.

Perfect! That works for podman at least

For future finders my file looks like

[engine]
static_dir = "/home/lkn/.local/share/containers/storage/libpod"
volume_path = "/home/lkn/.local/share/containers/storage/libpod"

Now trying to access existing containers with distrobox I get an error like

ERRO[0000] Unmounting /home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/merged: invalid argument 
Error: unable to start container "ff34fb71718140a638f7c834ed6d116f413d8a50f8bfbc88307a38d717f7ff52": mounting storage for container ff34fb71718140a638f7c834ed6d116f413d8a50f8bfbc88307a38d717f7ff52: creating overlay mount to /home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/merged, mount_data="lowerdir=/home/lkn/.local/share/containers/storage/overlay/l/6I7L4CUQJIQPFUYHFX5OZBL4JK:/home/lkn/.local/share/containers/storage/overlay/l/6I7L4CUQJIQPFUYHFX5OZBL4JK/../diff1:/home/lkn/.local/share/containers/storage/overlay/l/7JTTU5TJDPKSG76SE4CVJCD5OT:/home/lkn/.local/share/containers/storage/overlay/l/BFAWGXFYIDBJFEOEEW66NT5TUI,upperdir=/home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/diff,workdir=/home/lkn/.local/share/containers/storage/overlay/f79b472b5b98d8e7a728c7d52415ea60b1043ae60cb1d08ee766f8dc71c08963/work,userxattr,context=\"system_u:object_r:container_file_t:s0:c1022,c1023\"": invalid argument

But I'm happy to take this elsewhere if it's not directly related to podman 😄

EDIT: Managed to rescue, turns out it was SELinux. Turning it off but not fully disabled allowed me to carry on using my old containers; thanks @mheon you're a life-saver 😄

mheon added a commit to mheon/libpod that referenced this issue Dec 2, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
clemak27 added a commit to clemak27/linux_setup that referenced this issue Dec 3, 2023
clemak27 added a commit to clemak27/linux_setup that referenced this issue Dec 3, 2023
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/podman that referenced this issue Dec 3, 2023
When Podman starts, it checks a number of critical runtime paths
against stored values in the database to make sure that existing
containers are not broken by a configuration change. We recently
made some changes to this logic to make our handling of the some
options more sane (StaticDir in particular was set based on other
passed options in a way that was not particularly sane) which has
made the logic more sensitive to paths with symlinks. As a simple
fix, handle symlinks properly in our DB vs runtime comparisons.

The BoltDB bits are uglier because very, very old Podman versions
sometimes did not stuff a proper value in the database and
instead used the empty string. SQLite is new enough that we don't
have to worry about such things.

Fixes containers#20872

Signed-off-by: Matt Heon <mheon@redhat.com>
@nariba
Copy link

nariba commented Dec 7, 2023

Thank you very much!
I was having the same problem and this issue helped me solve it.

In my environment, there was no directory, not even containers.conf, but if I create the directory and file, it should be OK.
The containers.conf mentioned in this comment is in the following path,
/home/lkn/.config/containers/containers.conf

@debarshiray
Copy link
Member

debarshiray commented Dec 7, 2023

I couldn't reproduce this on my Fedora 39 Silverblue:

rishi@fedora:~$ podman version
Client:       Podman Engine
Version:      4.8.0
API Version:  4.8.0
Go Version:   go1.21.4
Built:        Tue Nov 28 10:58:32 2023
OS/Arch:      linux/amd64
rishi@fedora:~$ 
rishi@fedora:~$ echo $HOME
/var/home/rishi
rishi@fedora:~$ 

... but clearly a lot of people can. Based on these two comments, I think this affects those who have been upgrading their installations in-place for a while.

I usually reinstall once every year, and I remember that we changed the default value of HOME from /home/foo to /var/home/foo at some point to avoid relying on the symbolic link.

@walking-octopus
Copy link

stas@thinkpad-t14 /h/stas [1]> rpm-ostree status -b
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
BootedDeployment:
● fedora:fedora/39/x86_64/kinoite
                  Version: 39.20231205.0 (2023-12-05T01:04:12Z)
               BaseCommit: 694f995d4b7ca42e87b393ff3058949dad660e5f2d692338dacaaceab06cd754
             GPGSignature: Valid signature by E8F23996F23218640CB44CBE75CF5AC418B8E74C
          LayeredPackages: 'gcc-c++' adb alsa-lib-devel automake cmake duperemove ffmpeg-free fish gcc htop
                           jetbrains-mono-fonts-all make mozilla-openh264 podman-compose podman-docker
                           powertop systemd-devel tmux xeyes

AvailableUpdate:
           Diff: 7 upgraded
stas@thinkpad-t14 ~> echo $HOME
/home/stas
stas@thinkpad-t14 ~> export HOME=/var/home/stas
stas@thinkpad-t14 /h/stas> echo $HOME
/var/home/stas
stas@thinkpad-t14 /h/stas> podman --version
podman version 4.8.0
stas@thinkpad-t14 /h/stas> podman version
Error: database libpod root directory (staticdir) "/home/stas/.local/share/containers/storage/libpod" does not match our libpod root directory (staticdir) "/var/home/stas/.local/share/containers/storage/libpod": database configuration mismatch

Just changing the $HOME seems to not be enough. Perhaps there's a config file somewhere that I can edit?

It's also strange it didn't happen after the rebase but just another version update...

@elken
Copy link

elken commented Dec 8, 2023

Perhaps there's a config file somewhere that I can edit?

#20872 (comment)

@quazar-omega
Copy link

Does that workaround have any long-term consequences, or can we safely apply it temporarily until the fix lands?

@mheon
Copy link
Member

mheon commented Dec 8, 2023

There are no consequences even after the fix lands. The two paths are symlinked together, we're just forcing another version be used when doing database checks.

@quazar-omega
Copy link

Oh good, thanks for explaining!

@Justinzobel
Copy link

I rebased from Fedora 39 Kinoite to Rawhide Kinoite to do some testing and this affects my ability to run a toolbox:

toolbox
Error: failed to get the Podman version
...
podman --version
podman version 4.8.1

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Mar 17, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. regression
Projects
None yet
Development

Successfully merging a pull request may close this issue.