container: T4870: Update podman to use overlay storage driver

[TGNThump]

Change Summary

We currently use the vfs storage driver for podman, which has poor performance and results in much more disk space usage.

The vfs storage driver is intended for testing purposes, and for situations where no copy-on-write filesystem can be used. Performance of this storage driver is poor, and is not generally recommended for production use.

This PR switches the storage driver to overlay, which is docker's preferred storage driver.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, renaming)
• Refactoring (no functional changes)
• Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
• Other (please describe):

Related Task(s)

• https://phabricator.vyos.net/T4870

Component(s) name

• container

Proposed changes

How to test

Checklist:

• I have read the CONTRIBUTING document
• I have linked this PR to one or more Phabricator Task(s)
• I have run the components SMOKETESTS if applicable
• My commit headlines contain a valid Task id
• My change requires a change to the documentation
• I have updated the documentation accordingly

[sever-sever]

Only one question what to do if it exists old images/containers
As the new driver doesn't work with the old driver. So we should delete all container images/containers before we can use driver = overlay

[sarthurdev]

Good point, tested on system with existing vfs images:

ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve.  May prevent use of images created by other tools

Resolved by deleting all podman data with podman system reset.

[TGNThump]

Yeah, not sure what to do about the manual migration steps required.

[sarthurdev]

I don't think the image/container data can be converted between drivers.

I'm thinking it could either be an "expected breakage" as part of rolling releases, or a warning could be given on boot if vfs storage still exists: sudo podman info | grep "graphDriverName: vfs".

[c-po]

Would it be possible to move all the container storage files to a temporary place, run podman system reset and copy back the data? That could save us from numerous bug reports.

[sarthurdev]

The data would not be stored in a compatible format for the overlay storage driver.

[dmbaturin]

I believe the change is justified and that the best way to solve the compatibility is to provide a script that people can run before upgrade.

Since this feature is not in an LTS release, it's only used by the most adventurous users who are likely prepared to have to perform some manual migration steps.

[c-po]

The "incompatibility" can be resolved by running the following script prior to upgrading to a new image:

#!/bin/sh
# Migration helper script needed to be executed before upgrading to a more
# recent VyOS version as the container storage driver is changed incompatible

for pod in $(cli-shell-api listActiveNodes container name); do
    systemctl stop vyos-container-${pod//\'}.service
done

sed -i 's/vfs/overlay2/g' /etc/containers/storage.conf /usr/share/vyos/templates/container/storage.conf.j2
rm -rf /usr/lib/live/mount/persistence/container/storage/libpod

for pod in $(cli-shell-api listActiveNodes container name); do
    image=$(cli-shell-api returnActiveValue container name ${pod//\'} image)
    podman image pull $image
    systemctl start vyos-container-${pod//\'}.service
done

for dir in vfs vfs-containers vfs-images vfs-layers; do
    rm -rf /usr/lib/live/mount/persistence/container/storage/$dir
done

[c-po]

This manual step just got removed by f3956a5 which simply exports active images as oci-image and re-imports the image after the filesystem got migrated. No manual interaction by the user required.