[Bug]: Auto-update with pods may fail if there's only one container in the pod #17181
Labels
kind/bug
Categorizes issue or PR as related to a bug.
locked - please file new issue/PR
Assist humans wanting to comment on an old issue or PR with locked comments.
Comments
|
@vrothberg PTAL |
|
Thanks for reaching out, @saiarcot895. Could you share a reproducer? |
|
Sure, here's the systemd files for the pod and container:
This assumes that a subuid and subgid entry for Once this is set up, when there is an update, I do have two additional lines in both of the above files that I didn't include above: This is to reload my nftables rules. I'm assuming that this doesn't have an impact on the bug happening. |
|
@vrothberg Were you able to repro this issue? |
|
Using podman v4.4.0 (rootless) I am experiencing the same problem. After I build a new image locally, running |
|
@saiarcot895, I did not find time to look into the issue yet. I'll update this issue once I do. |
|
I have the same issue with linuxserver/bookstack pod. It never auto-updated itself. I have to manually pull latest image and restart pod in systemd. |
|
I think it also fails to update and rolls back from time to time in cases with more than one container in the pod. |
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 15, 2023
Relax the reverse dependency to the "parent" pod.service. This allows a container.service to be restarted without restarting the entire pod.service. Note that there is no system test for auto-updates with pods. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
|
Thanks everybody for the input and the patience. I opened #17508 to fix the issue. |
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 16, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 16, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
|
Note that I had to revisit the initial version of the PR. Auto updating a container inside a pod will always render the entire pod to be restarted - similar to what's happening in the |
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 16, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 16, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 16, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
vrothberg
added a commit
to vrothberg/libpod
that referenced
this issue
Feb 17, 2023
Support auto updating containers running inside pods. Similar to containers, the systemd units need to be generated via `podman-generate-systemd --new $POD` to generate the pod's units. Note that auto updating a container inside a pod will restart the entire pod. Updates of multiple containers inside a pod are batched, such that a pod is restarted at most once. That is effectively the same mechanism for auto updating containers in a K8s YAML via the `podman-kube@` template or via Quadlet. Updating a single container unit without restarting the entire pod is not possible. The reasoning behind is that pods are created with --exit-policy=stop which will render the pod to be stopped when auto updating the only container inside the pod. The (reverse) dependencies between the pod and its containers unit have been carefully selected for robustness. Changes may entail undesired side effects or backward incompatibilities that I am not comfortable with. Fixes: containers#17181 Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
github-actions
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Issue Description
My setup consists of multiple pods running one or more containers. All of the pods have user namespacing (or at least a UID/GID map) and a custom network specified. All containers have auto-update enabled. During the auto-update, if there's a container that needs to be updated, and that container is the only one running in that pod, then the auto-update might fail, because it seems that when the container is getting restarted by systemd, it's trying to bring down the pod at the same time. Therefore, the container gets rolled back.
Steps to reproduce the issue
Describe the results you received
The container was rolled back instead of getting updated.
Describe the results you expected
The container should've been updated successfully
podman info output
host: arch: amd64 buildahVersion: 1.28.0 cgroupControllers: - cpuset - cpu - io - memory - hugetlb - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon_2:2.1.5-0ubuntu22.04+obs14.22_amd64 path: /usr/bin/conmon version: 'conmon version 2.1.5, commit: ' cpuUtilization: idlePercent: 95.71 systemPercent: 0.86 userPercent: 3.43 cpus: 12 distribution: codename: jammy distribution: ubuntu version: "22.04" eventLogger: journald hostname: nuc idMappings: gidmap: null uidmap: null kernel: 5.19.4arcot-00008-g04e5cacdda90 linkmode: dynamic logDriver: journald memFree: 15378026496 memTotal: 66578608128 networkBackend: netavark ociRuntime: name: crun package: crun_1.7.2-0ubuntu22.04+obs48.12_amd64 path: /usr/bin/crun version: |- crun version 1.7.2 commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4 rundir: /run/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL os: linux remoteSocket: exists: true path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: false serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns_1.2.0-0ubuntu22.04+obs10.33_amd64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.3 swapFree: 3326406656 swapTotal: 5556404224 uptime: 691h 13m 17.00s (Approximately 28.79 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /usr/share/containers/storage.conf containerStore: number: 14 paused: 0 running: 14 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev,metacopy=on graphRoot: /var/lib/containers/storage graphRootAllocated: 493921239040 graphRootUsed: 258151227392 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "true" imageCopyTmpDir: /var/tmp imageStore: number: 13 runRoot: /run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 4.3.1 Built: 0 BuiltTime: Wed Dec 31 16:00:00 1969 GitCommit: "" GoVersion: go1.18.1 Os: linux OsArch: linux/amd64 Version: 4.3.1Podman in a container
No
Privileged Or Rootless
Privileged
Upstream Latest Release
Yes
Additional environment details
Ubuntu 22.04 with systemd 249
Additional information
Logs around the time of the update (logs from other containers have been excluded):
The text was updated successfully, but these errors were encountered: