Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Containers stop if dependency gets updated #18926

Closed
M1cha opened this issue Jun 19, 2023 · 1 comment · Fixed by #18929
Closed

Containers stop if dependency gets updated #18926

M1cha opened this issue Jun 19, 2023 · 1 comment · Fixed by #18929
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@M1cha
Copy link

M1cha commented Jun 19, 2023

Issue Description

I have two podman-systemd containers:

They both have:

  • auto start enabled using default.target
  • auto restart enabled using Restart=always
  • auto update enabled using Label=io.containers.autoupdate=registry

photoprism depends on photoprism-mariadb due to:

After=network.target photoprism-mariadb.service
Requires=photoprism-mariadb.service

Now, whenever photoprism-mariadb.service gets restarted because it was updated by podman-auto-update.service, photoprism stops until I start it manually. If I run systemctl restart photoprism-mariadb, photoprism will stop as well, but restart after 10 seconds as expected.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Create two containers in the given config
  2. Wait for an update to the dependency

Describe the results you received

The container stopped forever.

Describe the results you expected

The container should restart as expected.

podman info output

host:
  arch: arm64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc38.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 92.92
    systemPercent: 3.04
    userPercent: 4.04
  cpus: 6
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "38"
  eventLogger: journald
  hostname: homeserver
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.15-300.fc38.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 364490752
  memTotal: 3993206784
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.5-1.fc38.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-12.fc38.aarch64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 15985799168
  swapTotal: 17179865088
  uptime: 81h 40m 24.00s (Approximately 3.38 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 22
    paused: 0
    running: 20
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 491511971840
  graphRootUsed: 193108729856
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 48
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486872
  BuiltTime: Fri Apr 14 15:41:12 2023
  GitCommit: ""
  GoVersion: go1.20.2
  Os: linux
  OsArch: linux/arm64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

# cat /etc/containers/containers.conf

[containers]
userns = "auto"

Additional information

No response

@M1cha M1cha added the kind/bug Categorizes issue or PR as related to a bug. label Jun 19, 2023
@vrothberg
Copy link
Member

Thanks for reaching out, @M1cha!

I can reproduce. Internally podman auto-update does a stop + start instead of a restart which seems to be the source of the issue.

vrothberg added a commit to vrothberg/libpod that referenced this issue Jun 19, 2023
Commit f131eaa changed restart to a stop+start motivated by
comments in the systemd man pages that restart behaves different than
stop+start, for instance, that it keeps certain resources open and
treats timers differently.  Yet, the actually fix for containers#17607 in the very
same commit was dealing with an ENOENT of the CID file on container
removal.

As it turns out in in containers#18926, changing to stop+start regressed on
restarting dependencies when auto updating a systemd unit.  Hence, move
back to using restart to make sure that dependent systemd units are
restarted as well.

An alternative could be recommending to use `BindsTo=` in Quadlet files
but this seems less common than `Requires=` and hence more risky to
cause issues on user sites.

Fixes: containers#18926
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
cgiradkar pushed a commit to cgiradkar/podman that referenced this issue Jul 12, 2023
Commit f131eaa changed restart to a stop+start motivated by
comments in the systemd man pages that restart behaves different than
stop+start, for instance, that it keeps certain resources open and
treats timers differently.  Yet, the actually fix for containers#17607 in the very
same commit was dealing with an ENOENT of the CID file on container
removal.

As it turns out in in containers#18926, changing to stop+start regressed on
restarting dependencies when auto updating a systemd unit.  Hence, move
back to using restart to make sure that dependent systemd units are
restarted as well.

An alternative could be recommending to use `BindsTo=` in Quadlet files
but this seems less common than `Requires=` and hence more risky to
cause issues on user sites.

Fixes: containers#18926
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
cgiradkar pushed a commit to cgiradkar/podman that referenced this issue Jul 13, 2023
Commit f131eaa changed restart to a stop+start motivated by
comments in the systemd man pages that restart behaves different than
stop+start, for instance, that it keeps certain resources open and
treats timers differently.  Yet, the actually fix for containers#17607 in the very
same commit was dealing with an ENOENT of the CID file on container
removal.

As it turns out in in containers#18926, changing to stop+start regressed on
restarting dependencies when auto updating a systemd unit.  Hence, move
back to using restart to make sure that dependent systemd units are
restarted as well.

An alternative could be recommending to use `BindsTo=` in Quadlet files
but this seems less common than `Requires=` and hence more risky to
cause issues on user sites.

Fixes: containers#18926
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
cgiradkar pushed a commit to cgiradkar/podman that referenced this issue Jul 17, 2023
Commit f131eaa changed restart to a stop+start motivated by
comments in the systemd man pages that restart behaves different than
stop+start, for instance, that it keeps certain resources open and
treats timers differently.  Yet, the actually fix for containers#17607 in the very
same commit was dealing with an ENOENT of the CID file on container
removal.

As it turns out in in containers#18926, changing to stop+start regressed on
restarting dependencies when auto updating a systemd unit.  Hence, move
back to using restart to make sure that dependent systemd units are
restarted as well.

An alternative could be recommending to use `BindsTo=` in Quadlet files
but this seems less common than `Requires=` and hence more risky to
cause issues on user sites.

Fixes: containers#18926
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
cgiradkar pushed a commit to cgiradkar/podman that referenced this issue Jul 17, 2023
Commit f131eaa changed restart to a stop+start motivated by
comments in the systemd man pages that restart behaves different than
stop+start, for instance, that it keeps certain resources open and
treats timers differently.  Yet, the actually fix for containers#17607 in the very
same commit was dealing with an ENOENT of the CID file on container
removal.

As it turns out in in containers#18926, changing to stop+start regressed on
restarting dependencies when auto updating a systemd unit.  Hence, move
back to using restart to make sure that dependent systemd units are
restarted as well.

An alternative could be recommending to use `BindsTo=` in Quadlet files
but this seems less common than `Requires=` and hence more risky to
cause issues on user sites.

Fixes: containers#18926
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 18, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants