Inspect image of container for better idempotency #39

sshnaidm · 2020-05-10T21:00:06Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Podman container module should inspect image of container to understand better what are options will be set for container. It's required for better idempotency of podman_container module.

Steps to reproduce the issue:

For example if we have options set in Dockerfile/image like workdir, volume and any other supported, it'll be non-default in container inspection.

- containers.podman.podman_container:
    image: graphiteapp/graphite-statsd:latest
    name: graphite
    state: present

- containers.podman.podman_container:
    name: mysql
    image: mysql
    state: present
    pod: telemetry
    env:
      MYSQL_ROOT_PASSWORD: "2211"

For strict image idempotency we'll need to check SHAs of various images.

The text was updated successfully, but these errors were encountered:

jaudriga · 2020-05-20T11:39:00Z

Is there a recommeded workaround for this issue?

sshnaidm · 2020-05-20T16:48:52Z

@jaudriga actually it's an enhancement, that "nice to have". But idempotency with image should be fine now. Which issue do you hit with latest version?

jaudriga · 2020-05-25T13:00:24Z

@sshnaidm while idempotency with image may be fine, podman still recreates the container for the newest version of Graphite (where I set volume as well). I actually do not see a difference:

$ grep version /home/jrsr/.ansible/collections/ansible_collections/containers/podman/MANIFEST.json
  "version": "1.0.0",
$ ansible-playbook full.yml -CDv
...
TASK [graphite : Install Graphite container] *************************************************************************************************************************************************
--- before
+++ after
@@ -1,2 +1,2 @@
-image - graphite-statsd
-volume - ['/opt/graphite/conf:/opt/graphite/conf', '/opt/graphite/storage:/opt/graphite/storage', '027ddf42535de1ee53af893177e6a32170fd0c238a180e4cfea0778746bfb7f5:/opt/graphite/webapp/graphite/functions/custom', '0e5a5cf16d17f21ee38d91be6adc73d5f1ef77e3752b2f656e9ba2beeda0cfb9:/etc/logrotate.d', '3479cd25d0bc32d6f939cfeb1a8299f26a943e7b2102d1d3f3491c2654b767a4:/var/lib/redis', '5660a16982ded92a33754d10675a55ffbfb9cf801da2a6ee390ca01c0aaf8426:/etc/nginx', '9de5c5bfe685e5ae3c61487cdcb71615d36708c767a7607d67143f183cdfb334:/var/log', 'd3547bf26de911697ac434e574235703b85908603a6f208bbba21c76719b1076:/opt/statsd/config']
+image - graphite-statsd:1.1.7-2
+volume - ['/opt/graphite/conf:/opt/graphite/conf', '/opt/graphite/storage:/opt/graphite/storage']

I actually see that behavior for most of my containers (docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.9, graylog/graylog:3.3.0-1, mongo:4.0.18). Only grafana/grafana:7.0.0, seems to work well. But maybe we should stick to talking about graphite.

So, considering I am not doing anything wrong, my question is can I somehow use podman_container and set volume for graphite without it being recreated during every run?

sshnaidm · 2020-05-25T20:10:09Z

@jaudriga can you please paste your playbook for reproducing? Or at least the part with podman_container

jaudriga · 2020-05-26T07:39:00Z

@sshnaidm the following recreates the container on every run with the above diff:

- name: Install Graphite container
  containers.podman.podman_container:
    env:
      REDIS_TAGDB: 1
    image: "graphiteapp/graphite-statsd:1.1.7-2"
    name: graphite
    ports:
      - "80:80"
      - "2003-2004:2003-2004"
    state: present
    volumes:
      - "/opt/graphite/conf:/opt/graphite/conf"
      - "/opt/graphite/storage:/opt/graphite/storage"

jaudriga · 2020-06-02T12:40:32Z

@sshnaidm thanks for working on this so fast. Now.. it seems to handle locally mounted folders well. However, it does not seem to have resolved the issue with the image name:

$ ansible-playbook -CDv full.yml
TASK [graphite : Install Graphite container] *********************************************************************************************************************************************************************************************************************************************************************************************************
--- before
+++ after
@@ -1,2 +1,2 @@
-image - graphite-statsd
-volume - ['/opt/graphite/conf:/opt/graphite/conf']
+image - graphite-statsd:1.1.7-2
+volume - ['/opt/graphite/conf:/opt/graphite/conf', 'graphite-data:/opt/graphite/storage']

changed: [graphite.exp] => {"actions": ["recreated graphite"], "changed": true, "container": {"AppArmorProfile": "container-default-1.9.3", "Args": ["/entrypoint"], "BoundingCaps": ["CAP_AUDIT_WRITE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_MKNOD", "CAP_NET_BIND_SERVICE", "CAP_NET_RAW", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"], "Config": {"Annotations": {"io.container.manager": "libpod", "io.kubernetes.cri-o.Created": "2020-06-02T12:28:28.279713316Z", "io.kubernetes.cri-o.TTY": "false", "io.podman.annotations.autoremove": "FALSE", "io.podman.annotations.init": "FALSE", "io.podman.annotations.privileged": "FALSE", "io.podman.annotations.publish-all": "FALSE", "org.opencontainers.image.stopSignal": "1"}, "AttachStderr": false, "AttachStdin": false, "AttachStdout": false, "Cmd": null, "CreateCommand": ["podman", "container", "run", "--name", "graphite", "--env", "REDIS_TAGDB=1", "--publish", "80:80", "--publish", "2003-2004:2003-2004", "--volume", "/opt/graphite/conf:/opt/graphite/conf", "--volume", "graphite-data:/opt/graphite/storage", "--detach=True", "graphiteapp/graphite-statsd:1.1.7-2"], "Domainname": "", "Entrypoint": "/entrypoint", "Env": ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "TERM=xterm", "HOSTNAME=530686241e65", "container=podman", "STATSD_INTERFACE=udp", "REDIS_TAGDB=1", "HOME=/root"], "Hostname": "530686241e65", "Image": "docker.io/graphiteapp/graphite-statsd:latest", "Labels": {"maintainer": "Denys Zhdanov <denis.zhdanov@gmail.com>"}, "OnBuild": null, "OpenStdin": false, "StdinOnce": false, "StopSignal": 1, "Tty": false, "User": "", "Volumes": null, "WorkingDir": "/"}, "ConmonPidFile": "/var/run/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/conmon.pid", "Created": "2020-06-02T12:28:28.279713316Z", "Dependencies": [], "Driver": "overlay", "EffectiveCaps": ["CAP_AUDIT_WRITE", "CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_KILL", "CAP_MKNOD", "CAP_NET_BIND_SERVICE", "CAP_NET_RAW", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"], "ExecIDs": [], "ExitCommand": ["/usr/bin/podman", "--root", "/var/lib/containers/storage", "--runroot", "/var/run/containers/storage", "--log-level", "error", "--cgroup-manager", "systemd", "--tmpdir", "/var/run/libpod", "--runtime", "runc", "--events-backend", "file", "container", "cleanup", "530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119"], "GraphDriver": {"Data": {"LowerDir": "/var/lib/containers/storage/overlay/07eb7a8f8b30c8d9f2a2be1d3169b06b350f9e2bb87dcc4e8585ef8403e64d75/diff:/var/lib/containers/storage/overlay/4ef8d93c3f7e0fcafcb8e91640739f254030b78e780de8ae3bac8be8b1ab4008/diff:/var/lib/containers/storage/overlay/858f6f478edde3c923ec55b03da94412f35468d2e7771a594eed39bdd3b10a07/diff:/var/lib/containers/storage/overlay/beee9f30bc1f711043e78d4a2be0668955d4b761d587d6f60c2c8dc081efb203/diff", "MergedDir": "/var/lib/containers/storage/overlay/f5f8d30c6a376ae42a6505ac3bedb09a36f1020dcd12d538cf809693012ead4b/merged", "UpperDir": "/var/lib/containers/storage/overlay/f5f8d30c6a376ae42a6505ac3bedb09a36f1020dcd12d538cf809693012ead4b/diff", "WorkDir": "/var/lib/containers/storage/overlay/f5f8d30c6a376ae42a6505ac3bedb09a36f1020dcd12d538cf809693012ead4b/work"}, "Name": "overlay"}, "HostConfig": {"AutoRemove": false, "Binds": ["58f646ddf256925f996949bfa286b67e5dce61bab537495d9c760bf85fa4d619:/var/lib/redis:rprivate,rw,nodev,exec,nosuid,rbind", "504d0ecf1b68e612b62824618dd8a23f8b51e53a280a8d8e4bef22c0736b36ab:/var/log:rprivate,rw,nodev,exec,nosuid,rbind", "9db6dcc7d5734c0c1633543ce5511ccf7a501496231f1aab48c036cd96bed129:/etc/logrotate.d:rprivate,rw,nodev,exec,nosuid,rbind", "cde9a3e2e5491ea7313814f2ebc712d7ab48d7954b06ed48f6346c14aa652966:/etc/nginx:rprivate,rw,nodev,exec,nosuid,rbind", "graphite-data:/opt/graphite/storage:rw,rprivate,nosuid,nodev,rbind", "1a1cb9230b2075432cb13836164a5220e41ceb1ca1dfd5ba8808176bb7768039:/opt/graphite/webapp/graphite/functions/custom:rprivate,rw,nodev,exec,nosuid,rbind", "fd4e6a9f601231af729313e3c82a9efaee3fa0d375634cfdbc46883798e3110b:/opt/statsd/config:rprivate,rw,nodev,exec,nosuid,rbind", "/opt/graphite/conf:/opt/graphite/conf:rw,rprivate,rbind"], "BlkioDeviceReadBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceWriteIOps": null, "BlkioWeight": 0, "BlkioWeightDevice": null, "CapAdd": [], "CapDrop": [], "Cgroup": "", "CgroupParent": "", "Cgroups": "default", "ConsoleSize": [0, 0], "ContainerIDFile": "", "CpuCount": 0, "CpuPercent": 0, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpuShares": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DiskQuota": 0, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": [], "GroupAdd": [], "IOMaximumBandwidth": 0, "IOMaximumIOps": 0, "IpcMode": "", "Isolation": "", "KernelMemory": 0, "Links": null, "LogConfig": {"Config": null, "Type": "k8s-file"}, "Memory": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": -1, "NanoCpus": 0, "NetworkMode": "default", "OomKillDisable": false, "OomScoreAdj": 0, "PidMode": "", "PidsLimit": 4096, "PortBindings": {"2003/tcp": [{"HostIp": "", "HostPort": "2003"}], "2004/tcp": [{"HostIp": "", "HostPort": "2004"}], "80/tcp": [{"HostIp": "", "HostPort": "80"}]}, "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "RestartPolicy": {"MaximumRetryCount": 0, "Name": ""}, "Runtime": "oci", "SecurityOpt": [], "ShmSize": 65536000, "Tmpfs": {}, "UTSMode": "", "Ulimits": [{"Hard": 1048576, "Name": "RLIMIT_NOFILE", "Soft": 1048576}, {"Hard": 32768, "Name": "RLIMIT_NPROC", "Soft": 32768}], "UsernsMode": "", "VolumeDriver": "", "VolumesFrom": null}, "HostnamePath": "/var/run/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/hostname", "HostsPath": "/var/run/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/hosts", "Id": "530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119", "Image": "0869fbfe98ecbb9438e53c9bc733fcb2e73b652d519fe04912bae0f69126abf2", "ImageName": "docker.io/graphiteapp/graphite-statsd:latest", "IsInfra": false, "LogPath": "/var/lib/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/ctr.log", "LogTag": "", "MountLabel": "", "Mounts": [{"Destination": "/var/lib/redis", "Driver": "local", "Mode": "", "Name": "58f646ddf256925f996949bfa286b67e5dce61bab537495d9c760bf85fa4d619", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/58f646ddf256925f996949bfa286b67e5dce61bab537495d9c760bf85fa4d619/_data", "Type": "volume"}, {"Destination": "/var/log", "Driver": "local", "Mode": "", "Name": "504d0ecf1b68e612b62824618dd8a23f8b51e53a280a8d8e4bef22c0736b36ab", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/504d0ecf1b68e612b62824618dd8a23f8b51e53a280a8d8e4bef22c0736b36ab/_data", "Type": "volume"}, {"Destination": "/etc/logrotate.d", "Driver": "local", "Mode": "", "Name": "9db6dcc7d5734c0c1633543ce5511ccf7a501496231f1aab48c036cd96bed129", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/9db6dcc7d5734c0c1633543ce5511ccf7a501496231f1aab48c036cd96bed129/_data", "Type": "volume"}, {"Destination": "/etc/nginx", "Driver": "local", "Mode": "", "Name": "cde9a3e2e5491ea7313814f2ebc712d7ab48d7954b06ed48f6346c14aa652966", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/cde9a3e2e5491ea7313814f2ebc712d7ab48d7954b06ed48f6346c14aa652966/_data", "Type": "volume"}, {"Destination": "/opt/graphite/storage", "Driver": "local", "Mode": "", "Name": "graphite-data", "Options": ["nosuid", "nodev", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/graphite-data/_data", "Type": "volume"}, {"Destination": "/opt/graphite/webapp/graphite/functions/custom", "Driver": "local", "Mode": "", "Name": "1a1cb9230b2075432cb13836164a5220e41ceb1ca1dfd5ba8808176bb7768039", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/1a1cb9230b2075432cb13836164a5220e41ceb1ca1dfd5ba8808176bb7768039/_data", "Type": "volume"}, {"Destination": "/opt/statsd/config", "Driver": "local", "Mode": "", "Name": "fd4e6a9f601231af729313e3c82a9efaee3fa0d375634cfdbc46883798e3110b", "Options": ["nodev", "exec", "nosuid", "rbind"], "Propagation": "rprivate", "RW": true, "Source": "/var/lib/containers/storage/volumes/fd4e6a9f601231af729313e3c82a9efaee3fa0d375634cfdbc46883798e3110b/_data", "Type": "volume"}, {"Destination": "/opt/graphite/conf", "Driver": "", "Mode": "", "Name": "", "Options": ["rbind"], "Propagation": "rprivate", "RW": true, "Source": "/opt/graphite/conf", "Type": "bind"}], "Name": "graphite", "Namespace": "", "NetworkSettings": {"Bridge": "", "EndpointID": "", "Gateway": "10.88.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "HairpinMode": false, "IPAddress": "10.88.0.191", "IPPrefixLen": 16, "IPv6Gateway": "", "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "MacAddress": "d6:bf:94:bb:31:f8", "Ports": [{"containerPort": 80, "hostIP": "", "hostPort": 80, "protocol": "tcp"}, {"containerPort": 2003, "hostIP": "", "hostPort": 2003, "protocol": "tcp"}, {"containerPort": 2004, "hostIP": "", "hostPort": 2004, "protocol": "tcp"}], "SandboxID": "", "SandboxKey": "/var/run/netns/cni-2c999ba1-e517-473f-a9e3-b2c3cf52ae87"}, "OCIConfigPath": "/var/lib/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/config.json", "OCIRuntime": "runc", "Path": "/entrypoint", "Pod": "", "ProcessLabel": "", "ResolvConfPath": "/var/run/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata/resolv.conf", "RestartCount": 0, "Rootfs": "", "State": {"ConmonPid": 12840, "Dead": false, "Error": "", "ExitCode": 0, "FinishedAt": "0001-01-01T00:00:00Z", "Healthcheck": {"FailingStreak": 0, "Log": null, "Status": ""}, "OOMKilled": false, "OciVersion": "1.0.1-dev", "Paused": false, "Pid": 12879, "Restarting": false, "Running": true, "StartedAt": "2020-06-02T12:28:28.784235249Z", "Status": "running"}, "StaticDir": "/var/lib/containers/storage/overlay-containers/530686241e65b916af9505f256772b7d88a0c51cc2337dc2a7d0c70d616d6119/userdata"}, "podman_actions": ["podman rm -f graphite", "podman run --name graphite --env REDIS_TAGDB=1 --publish 80:80 --publish 2003-2004:2003-2004 --volume /opt/graphite/conf:/opt/graphite/conf --volume graphite-data:/opt/graphite/storage --detach=True graphiteapp/graphite-statsd:1.1.7-2"], "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

(In the meantime I switched from mounting the storage folder on the local filesystem to using a volume I created via podman volume create --opt device={{ data_device }} graphite-data. It does not seem to like this either. However, this might be considered a separate issue)

The following is the new code:

- name: Install Graphite container
  containers.podman.podman_container:
    env:
      REDIS_TAGDB: 1
    image: "graphiteapp/graphite-statsd:{{ service_versions.dict.graphite.version }}"
    name: graphite
    ports:
      - "80:80"
      - "2003-2004:2003-2004"
    state: present
    volumes:
      - "/opt/graphite/conf:/opt/graphite/conf"
      - "graphite-data:/opt/graphite/storage"

sshnaidm · 2020-06-03T17:46:44Z

@jaudriga thanks, you provide very good cases :) Will look asap.

sshnaidm · 2020-06-05T12:06:19Z

@jaudriga let me understand please the image issue better, because it's not reproducible for me:

-image - graphite-statsd
+image - graphite-statsd:1.1.7-2

What is image tag in first run and what is it in second one?
And what is value of {{ service_versions.dict.graphite.version }}?

jaudriga · 2020-06-08T07:06:05Z

it's not reproducible for me

@sshnaidm The issue seems to be resolved for me as well. Yeah :) . However, I am going to answer your questions in case anyone else will have a similar issue.

What is image tag in first run and what is it in second one?

The image tag was the same in the first and the second run: 1.1.7-2. I remember that, before the issue was fixed, when I ran podman inspect Image or ImageName did not mention the tag 1.1.7-2 as it does now:

# podman inspect graphite | grep graphite-statsd
        "ImageName": "docker.io/graphiteapp/graphite-statsd:1.1.7-2",
            "Image": "docker.io/graphiteapp/graphite-statsd:1.1.7-2",
                "graphiteapp/graphite-statsd:1.1.7-2"

Sadly I did not document the output of podman inspect at the time.

And what is value of {{ service_versions.dict.graphite.version }}?

Arg. Sry I forgot to replace the above service_versions variable. Its value is 1.1.7-2.

sshnaidm · 2020-06-08T08:47:36Z

@jaudriga cool, I hope last patch solved it for you. Let me know please if there are any other issues.
Thanks!

jaudriga · 2020-06-09T07:11:25Z

@sshnaidm I just tried out version 1.0.3 of this collection. It is working fine! Thank you :) .

I noticed something of a follow-up issue for which I currently do not want to create a separate issue.. just wanted to let you know:

When using

- name: Update Graylog Container
   containers.podman.podman_container:
     volumes:
       - "graylog-journal:/usr/share/graylog/data/journal"
       - "graylog-plugin:/usr/share/graylog/plugin/"

The diff showed the following reproducibly:

$ ansible-playbook --vault-password-file password.txt full.yml -CD
...
TASK [graylog : Update Graylog Container] ************************************************************************************************************************************************************************************************************************************************************************************************************
--- before
+++ after
@@ -1 +1 @@
-volume - []
+volume - ['graylog-plugin:/usr/share/graylog/plugin/']

changed: [logging.exp]

However, removing the trailing / fixed this.

sshnaidm · 2020-06-11T17:04:55Z

@jaudriga yes, podman remove all slashes in inspect. So we need to strip all ending slashes in input too. Will send a fix.

sshnaidm added bug Something isn't working bug/idempotency Bug related to idempotency of modules labels May 10, 2020

sshnaidm self-assigned this May 10, 2020

sshnaidm mentioned this issue May 26, 2020

Inspect image for better idempotency #49

Merged

sshnaidm closed this as completed in #49 May 31, 2020

sshnaidm reopened this Jun 3, 2020

sshnaidm mentioned this issue Jun 5, 2020

Add idempotency for existing local volumes #56

Merged

sshnaidm closed this as completed in #56 Jun 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Inspect image of container for better idempotency #39

Inspect image of container for better idempotency #39

sshnaidm commented May 10, 2020 •

edited

jaudriga commented May 20, 2020

sshnaidm commented May 20, 2020

jaudriga commented May 25, 2020

sshnaidm commented May 25, 2020

jaudriga commented May 26, 2020 •

edited

jaudriga commented Jun 2, 2020

sshnaidm commented Jun 3, 2020

sshnaidm commented Jun 5, 2020 •

edited

jaudriga commented Jun 8, 2020

sshnaidm commented Jun 8, 2020

jaudriga commented Jun 9, 2020

sshnaidm commented Jun 11, 2020

Inspect image of container for better idempotency #39

Inspect image of container for better idempotency #39

Comments

sshnaidm commented May 10, 2020 • edited

jaudriga commented May 20, 2020

sshnaidm commented May 20, 2020

jaudriga commented May 25, 2020

sshnaidm commented May 25, 2020

jaudriga commented May 26, 2020 • edited

jaudriga commented Jun 2, 2020

sshnaidm commented Jun 3, 2020

sshnaidm commented Jun 5, 2020 • edited

jaudriga commented Jun 8, 2020

sshnaidm commented Jun 8, 2020

jaudriga commented Jun 9, 2020

sshnaidm commented Jun 11, 2020

sshnaidm commented May 10, 2020 •

edited

jaudriga commented May 26, 2020 •

edited

sshnaidm commented Jun 5, 2020 •

edited