Quadlet: Container with `Notify=healthy` does not respect `HealthStartPeriod` larger than systemd `TimeoutStartSec`

[l-austenfeld]

Issue Description

When starting a container via quadlet with the setting Notify=healthy, values for HealthStartPeriod larger than the default systemd TimeoutStartSec (default: 90s) are not respected.
This is because with Notify=healthy systemd will see the service as starting until the startup heathcheck succeeds. So after it sees the service starting for longer than its own startup timeout it will stop the service and mark it as failed (systemd logs systemd[1]: healthcheck.service: start operation timed out. Terminating. and systemd[1]: healthcheck.service: Failed with result 'timeout'.).

This issue can be fixed by setting TimeoutStartSec on the service. Either to infinity to disable the timeout in systemd or to HealthStartPeriod plus some extra time to account for image pulls, container creation, etc.

Steps to reproduce the issue

• Add this quadlet file to /etc/containers/systemd:

[Container]
Image=docker.io/library/alpine:latest
Exec=/bin/sh -c "sleep 180; touch /startup_done; sleep 999999"

Notify=healthy
HealthStartupRetries=0
HealthStartPeriod=20m

HealthStartupCmd=["/usr/bin/test", "-f", "/startup_done"]
HealthStartupInterval=10s
HealthStartupTimeout=2s
HealthCmd=["/usr/bin/test", "-f", "/startup_done"]
HealthInterval=1m
HealthTimeout=2s

# Uncomment this to fix the issue
# [Service]
# TimeoutStartSec=infinity

• sudo systemctl daemon-reload
• sudo systemctl start healthcheck.service
• Wait for it to fail
• Optional: Check /run/systemd/generator/healthcheck.service to see no generated TimeoutStartSec setting
• Uncomment the [Service] section in the quadlet file
• sudo systemctl daemon-reload
• sudo systemctl start healthcheck.service
• Wait for it to start successfully

Describe the results you received

The service fails to start

# systemctl status healthcheck.service
× healthcheck.service
     Loaded: loaded (/etc/containers/systemd/healthcheck.container; generated)
     Active: failed (Result: timeout) since Tue 2025-10-14 12:30:29 CEST; 30s ago

Describe the results you expected

The service starts without problem

# systemctl status healthcheck.service
● healthcheck.service
     Loaded: loaded (/etc/containers/systemd/healthcheck.container; generated)
     Active: active (running) since Tue 2025-10-14 12:35:12 CEST; 6s ago

podman info output

host:
  arch: amd64
  buildahVersion: 1.41.5
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 97.58
    systemPercent: 0.65
    userPercent: 1.76
  cpus: 24
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  emulatedArchitectures:
  - linux/arm
  - linux/arm64
  - linux/arm64be
  - linux/loong64
  - linux/mips
  - linux/mips64
  - linux/ppc
  - linux/ppc64
  - linux/ppc64le
  - linux/riscv32
  - linux/riscv64
  - linux/s390x
  eventLogger: journald
  freeLocks: 2046
  hostname: deathstar
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
  kernel: 6.17.2-273-tkg-eevdf-llvm
  linkmode: dynamic
  logDriver: journald
  memFree: 22459785216
  memTotal: 33566597120
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.16.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.16.0
    package: netavark-1.16.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.16.1
  ociRuntime:
    name: crun
    package: crun-git-1.21.r141.g9489b996-1
    path: /usr/bin/crun
    version: |-
      crun version 1.21.0.0.0.141-9489
      commit: 9489b9962b2fa40e7c6b5d7260f40b3baa3cbd89
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_09_19.623dbf6-1
    version: |
      pasta 2025_09_19.623dbf6
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.3-1
    version: |-
      slirp4netns version 1.3.3
      commit: 944fa94090e1fd1312232cbc0e6b43585553d824
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.6
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 0h 5m 16.00s
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  docker.io:
    Blocked: false
    Insecure: false
    Location: docker.io
    MirrorByDigestOnly: false
    Mirrors:
    - Insecure: false
      Location: docker-cache.***
      PullFromMirror: ""
    Prefix: docker.io
    PullFromMirror: ""
store:
  configFile: /home/***/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/***/.local/share/containers/storage
  graphRootAllocated: 999173910528
  graphRootUsed: 813741178880
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/***/.local/share/containers/storage/volumes
version:
  APIVersion: 5.6.2
  Built: 1759398423
  BuiltTime: Thu Oct  2 11:47:03 2025
  GitCommit: 9dd5e1ed33830612bc200d7a13db00af6ab865a4
  GoVersion: go1.25.1 X:nodwarf5
  Os: linux
  OsArch: linux/amd64
  Version: 5.6.2

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Tested an Archlinux using distro podman (see info output above) reproduced on a Debian Trixie VM using git version 32d6c54

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[mheon]

@ygalblum PTAL. It seems like it could be reasonable to error on health startup period longer than TimeoutStartSeconds?

[l-austenfeld]

It seems like it could be reasonable to error on health startup period longer than TimeoutStartSeconds?

Personal note: When I set HealthStartPeriod I have already expressed the desire to allow the container a certain time to start up. When that did not work, I kept looking for other quadlet options that I may need to set to allow the container to start fully (example: HealthStartupRetries is also needed). I would have expected quadlet to set the systemd TimeoutStartSeconds option.

Technical note: The default value for TimeoutStartSeconds is configurable via DefaultTimeoutStartSec in systemd-system.conf which has various locations for config and drop-ins. Determining the correct value to compare to may be complicated.

[Luap99]

Technical note: The default value for TimeoutStartSeconds is configurable via DefaultTimeoutStartSec in systemd-system.conf which has various locations for config and drop-ins. Determining the correct value to compare to may be complicated.

correct that is impossible more or less in the scope of quadlet

What we did for image pull is to use EXTEND_TIMEOUT_USEC on the notify socket which is much nicer then overwriting TimeoutStartSeconds=

Because that way we can update it dynamically only if we actually wait for the healthcheck, if there is a bug and we are let's say deadlocked in a different code part it doesn't have to wait so long and can use the shorter timeout value. So I suggest we do the same here.

https://github.com/containers/container-libs/blob/a3b0f19c3ea12cb02f0b11788489c52d4fb02e96/common/libimage/copier.go#L392-L421

[l-austenfeld]

I did not know about EXTEND_TIMEOUT_USEC, that does sound like a much better solution.

With a quick search I found the waitForHealthy function, which waits for the container to be healthy and then sends the final READY=1 message. That or the WaitForConditionWithInterval function it calls look like good candidates for adding that functionality to me.

[ygalblum]

@Luap99 so, IIUC your suggestion is a fix inside Podman, not in Quadlet. Right?

[Luap99]

@Luap99 so, IIUC your suggestion is a fix inside Podman, not in Quadlet. Right?

correct that would go into the podman code path

[Mujib-Ahasan]

So far I understand: with Notify=healthy, the larger value of HealthStartPeriod getting ineffective because startup healthcheck is taking more than default time (TimeoutStartSec ). But instead of dynamically overwrite TimeoutStartSec, we can use EXTEND_TIMEOUT_USEC in waitForHealthy or WaitForConditionWithInterval. Am I correct?? or missing something?

[Mujib-Ahasan]

/assign