systemd unit with --new fails to control or update the container #7285

IMetZach · 2020-08-10T20:09:07Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Steps to reproduce the issue:

Deploy my container as root:

# podman create -d --name pihole --label "io.containers.autoupdate=image" --network host -e TZ="America/New York" --privileged -v "/opt/pihole/etc-pihole/:/etc/pihole/:z" -v "/opt/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/:z" --dns=127.0.0.1 --dns=1.1.1.1 --hostname pi.hole -e VIRTUAL_HOST="pi.hole" -e PROXY_LOCATION="pi.hole" -e ServerIP="127.0.0.1" pihole/pihole:latest

Generate a systemd unit file:

# podman generate systemd --new --files pihole
/root/container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service
# mv /root/container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service /etc/systemd/system/container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service
# systemctl daemon-reload

Start the container.
Notice the failure to start

# systemctl start container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service
Job for container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service failed because the control process exited with error code.
See "systemctl status container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service" and "journalctl -xe" for details.

Check the output of the journal:

# journalctl -xe -u container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service
-- The job identifier is 15188 and the job result is done.
Aug 10 13:47:28 pihole.redziput.lan systemd[1]: container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service: Start request repeated too quickly.
Aug 10 13:47:28 pihole.redziput.lan systemd[1]: container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service has entered the 'failed' state with result 'exit-code'.
Aug 10 13:47:28 pihole.redziput.lan systemd[1]: Failed to start Podman container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service.
-- Subject: A start job for unit container-28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Attempt to use the auto-update feature of podman:

# podman auto-update
Trying to pull docker://pihole/pihole:latest...
Getting image source signatures
Copying blob 0f00de0e80bf skipped: already exists
Copying blob a42b41aa464a skipped: already exists
Copying blob 0e261c69a668 skipped: already exists
Copying blob 018c74957032 skipped: already exists
Copying blob c408adf5aab3 skipped: already exists
Copying blob e1760851e187 skipped: already exists
Copying blob fdc2ebbe8553 skipped: already exists
Copying blob 11921923a10e [--------------------------------------] 0.0b / 0.0b
Copying config 58c7e408f4 done
Writing manifest to image destination
Storing signatures
Error: 1 error occurred:
        * error auto-updating container "28850079097aa28a5bedd61c7f0b03b0a66b1fbe67f90179805e9fcd04ce23e8": no PODMAN_SYSTEMD_UNIT label found

Describe the results you received:
A systemd unit that does not start / update the container

Describe the results you expected:
I'm expecting the systemd unit to start the container and automatically update it to the new image.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      2.0.4
API Version:  1
Go Version:   go1.14.6
Built:        Wed Dec 31 19:00:00 1969
OS/Arch:      linux/arm64

$ rpm-ostree status
State: idle
Deployments:
● ostree://fedora-iot:fedora/stable/aarch64/iot
                   Version: 32.20200805.0 (2020-08-05T18:25:53Z)
                    Commit: 7d61df1f1ca8dcc9bddb97decddc162b1342121f1ccbcdefbd406235c17736ea
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4

Output of podman info --debug:

$ podman info --debug
host:
  arch: arm64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 057d6daf09d94e4febd5633a820aee1508c8366c'
  cpus: 4
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: pihole.redziput.lan
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.11-200.fc32.aarch64
  linkmode: dynamic
  memFree: 97943552
  memTotal: 983441408
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.aarch64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 467996672
  swapTotal: 467996672
  uptime: 3h 26m 45.53s (Approximately 0.12 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /var/home/elliott/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.aarch64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /var/home/elliott/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/elliott/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 19:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/arm64
  Version: 2.0.4

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-2.0.4-1.fc32.aarch64

Additional environment details (AWS, VirtualBox, physical, etc.):
Raspberry Pi 3 running Fedora IoT

The text was updated successfully, but these errors were encountered:

vrothberg · 2020-08-11T08:07:14Z

Thanks for opening the issue! Let's figure out why the generated unit fails. Auto-update won't work when the container wasn't started by systemd.

Could you regenerate the unit with --restart-policy=no and share the logs? We need to get the Podman logs to see why it has failed. They're currently masked by "Start request repeated too quickly."

IMetZach · 2020-08-11T14:37:07Z

Here is the output after it fails with the --restart-policy=no added to it:

-- The job identifier is 30613 and the job result is failed.
Aug 11 10:27:15 pihole.redziput.lan systemd[1]: Starting Podman container-b8375ed2da84069a029539cc4c3d1d996159573ae6e11160111a6b8c74849ab7.service...
-- Subject: A start job for unit container-b8375ed2da84069a029539cc4c3d1d996159573ae6e11160111a6b8c74849ab7.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit container-b8375ed2da84069a029539cc4c3d1d996159573ae6e11160111a6b8c74849ab7.service has begun execution.
--
-- The job identifier is 71717.
Aug 11 10:27:16 pihole.redziput.lan podman[65866]: Error: repository name must be lowercase
Aug 11 10:27:16 pihole.redziput.lan systemd[1]: container-b8375ed2da84069a029539cc4c3d1d996159573ae6e11160111a6b8c74849ab7.service: Control process exited, code=exited, status=125/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- An ExecStart= process belonging to unit container-b8375ed2da84069a029539cc4c3d1d996159573ae6e11160111a6b8c74849ab7.service has exited.

I noticed the part Error: repository name must be lowercase so I inspected the container and here is the create command:

# podman inspect pihole
...
            "CreateCommand": [
                "podman",
                "create",
                "-d",
                "--name",
                "pihole",
                "--label",
                "io.containers.autoupdate=image",
                "--network",
                "host",
                "-e",
                "TZ=America/New York",
                "--privileged",
                "-v",
                "/opt/pihole/etc-pihole/:/etc/pihole/:z",
                "-v",
                "/opt/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/:z",
                "--dns=127.0.0.1",
                "--dns=1.1.1.1",
                "--hostname",
                "pi.hole",
                "-e",
                "VIRTUAL_HOST=pi.hole",
                "-e",
                "PROXY_LOCATION=pi.hole",
                "-e",
                "ServerIP=127.0.0.1",
                "pihole/pihole:latest"
            ]

I'm wondering if having the arguments in caps are throwing it off.

IMetZach · 2020-08-11T14:49:18Z

Ok. I tried redeploying the container with all lower case arguments using the following command:

   podman create -d --name pihole \
--label "io.containers.autoupdate=image" \
--network host \
-e tz="america/new york" \
--privileged \
-v "/opt/pihole/etc-pihole/:/etc/pihole/:z" \
-v "/opt/pihole/etc-dnsmasq.d/:/etc/dnsmasq.d/:z" \
--dns=127.0.0.1 \
--dns=1.1.1.1 \
--hostname pi.hole \
-e virtual_host="pi.hole" \
-e proxy_location="pi.hole" \
-e serverip="127.0.0.1" \
pihole/pihole:latest

It failed and when I went digging through the journals I discovered this error over and over:

Aug 11 10:43:04 pihole.redziput.lan systemd[1]: Starting Podman container-0d3ca53a610848d9778bb98ad9e9f3a4f6be63c71194c240f8a53f84e55329f3.service...
-- Subject: A start job for unit container-pihole.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit container-pihole.service has begun execution.
--
-- The job identifier is 74067.
Aug 11 10:43:04 pihole.redziput.lan podman[68979]: Trying to pull registry.fedoraproject.org/york...
Aug 11 10:43:05 pihole.redziput.lan podman[68979]:   manifest unknown: manifest unknown
Aug 11 10:43:05 pihole.redziput.lan podman[68979]: Trying to pull registry.access.redhat.com/york...
Aug 11 10:43:05 pihole.redziput.lan podman[68979]:   name unknown: Repo not found
Aug 11 10:43:05 pihole.redziput.lan podman[68979]: Trying to pull registry.centos.org/york...
Aug 11 10:43:05 pihole.redziput.lan podman[68979]:   manifest unknown: manifest unknown
Aug 11 10:43:05 pihole.redziput.lan podman[68979]: Trying to pull docker.io/library/york...
Aug 11 10:43:06 pihole.redziput.lan podman[68979]:   denied: requested access to the resource is denied
Aug 11 10:43:06 pihole.redziput.lan podman[68979]: Error: unable to pull york: 4 errors occurred:
Aug 11 10:43:06 pihole.redziput.lan podman[68979]:         * Error initializing source docker://registry.fedoraproject.org/york:latest: Error reading manifest latest in registry.fedoraproject.org/york: manife>
Aug 11 10:43:06 pihole.redziput.lan podman[68979]:         * Error initializing source docker://registry.access.redhat.com/york:latest: Error reading manifest latest in registry.access.redhat.com/york: name u>
Aug 11 10:43:06 pihole.redziput.lan podman[68979]:         * Error initializing source docker://registry.centos.org/york:latest: Error reading manifest latest in registry.centos.org/york: manifest unknown: ma>
Aug 11 10:43:06 pihole.redziput.lan podman[68979]:         * Error initializing source docker://york:latest: Error reading manifest latest in docker.io/library/york: errors:
Aug 11 10:43:06 pihole.redziput.lan podman[68979]: denied: requested access to the resource is denied
Aug 11 10:43:06 pihole.redziput.lan podman[68979]: unauthorized: authentication required
Aug 11 10:43:06 pihole.redziput.lan systemd[1]: container-pihole.service: Control process exited, code=exited, status=125/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- An ExecStart= process belonging to unit container-pihole.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 125.

It appears that it is reading the york in "New York" as the name of the image and not the final argument.

IMetZach · 2020-08-11T15:58:41Z

After making it all lowercase and removing the argument -e tz="america/new york" the auto-update works fine.

vrothberg · 2020-08-18T08:48:32Z

Sorry for not coming back earlier. There's an issue when storing the arguments. I'll look into a fix now.

Make sure that arguments with whitespace are properly quoted so they are interpreted as one (and not multiple ones) by systemd. Now `-e tz="america/new york"` will be generated as `-e "tz=america/new york"`. The quotes are moving but the argument is still correct. Fixes: containers#7285 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

vrothberg · 2020-08-18T09:34:21Z

Opened #7350 to address the issue.

Make sure that arguments with whitespace are properly quoted so they are interpreted as one (and not multiple ones) by systemd. Now `-e tz="america/new york"` will be generated as `-e "tz=america/new york"`. The quotes are moving but the argument is still correct. Fixes: containers#7285 Signed-off-by: Valentin Rothberg <rothberg@redhat.com>

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 10, 2020

vrothberg self-assigned this Aug 11, 2020

vrothberg mentioned this issue Aug 18, 2020

generate systemd: quote arguments with whitespace #7350

Merged

IMetZach closed this as completed Aug 18, 2020

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

systemd unit with --new fails to control or update the container #7285

systemd unit with --new fails to control or update the container #7285

IMetZach commented Aug 10, 2020

vrothberg commented Aug 11, 2020

IMetZach commented Aug 11, 2020

IMetZach commented Aug 11, 2020 •

edited

IMetZach commented Aug 11, 2020 •

edited

vrothberg commented Aug 18, 2020

vrothberg commented Aug 18, 2020

systemd unit with --new fails to control or update the container #7285

systemd unit with --new fails to control or update the container #7285

Comments

IMetZach commented Aug 10, 2020

vrothberg commented Aug 11, 2020

IMetZach commented Aug 11, 2020

IMetZach commented Aug 11, 2020 • edited

IMetZach commented Aug 11, 2020 • edited

vrothberg commented Aug 18, 2020

vrothberg commented Aug 18, 2020

IMetZach commented Aug 11, 2020 •

edited

IMetZach commented Aug 11, 2020 •

edited