podman docker-compose healthchecks issue

[pclass-sensonix]

I am running podman 3.1.2-dev on a custom embedded Linux device. I have a docker-compose.yml file with an application named 'app1', which is a simple Python3 Flask webserver. Podman successfully brings up the docker-compose environment, but my healthchecks always return error, regardless of what I use for timeouts, intervals, retries, etc. My defined healthcheck works when I exec into the container and run them manually. When I inspect the application with 'podman inspect app1', I can see 2 'CMD-SHELL' instances defined. I believe it should only have mine and perhaps this is the issue.

Output of podman inspect app1:

"Healthcheck": {
    "Test": [
        "CMD-SHELL",
        "CMD-SHELL /usr/bin/wget http://127.0.0.1 || exit 1"
    ],
    "StartPeriod": 300000000000,
    "Interval": 30000000000,
    "Timeout": 60000000000,
    "Retries": 5
},

"Healthcheck": {
        "Status": "unhealthy",
        "FailingStreak": 21,
        "Log": [
            {
                "Start": "2021-06-09T14:53:29.192875476Z",
                "End": "2021-06-09T14:53:31.211062949Z",
                "ExitCode": 1,
                "Output": ""
            },
            {
                "Start": "2021-06-09T14:54:07.933818178Z",
                "End": "2021-06-09T14:54:10.368276033Z",
                "ExitCode": 1,
                "Output": ""
            },
            {
                "Start": "2021-06-09T14:54:47.115168651Z",
                "End": "2021-06-09T14:54:49.119256984Z",
                "ExitCode": 1,
                "Output": ""
            },
            {
                "Start": "2021-06-09T14:55:26.183508777Z",
                "End": "2021-06-09T14:55:28.038064193Z",
                "ExitCode": 1,
                "Output": ""
            },
            {
                "Start": "2021-06-09T14:56:02.115129058Z",
                "End": "2021-06-09T14:56:03.41780082Z",
                "ExitCode": 1,
                "Output": ""
            }
        ]
    }
},

Manually running healthchecks:

root@robot:~# podman healthcheck run app1
unhealthy
root@robot:~# 
root@robot:~# podman exec -it app1 sh
/ # 
/ # /usr/bin/wget http://127.0.0.1 || exit 1
Connecting to 127.0.0.1 (127.0.0.1:80)
saving to 'index.html'
index.html           100% |*******************************************************************************|    39  0:00:00 ETA
'index.html' saved
/ # 
/ # echo $?
0
/ # exit
root@robot:~# 

root@robot:~# podman --version
podman version 3.2.1-dev
root@robot:~# 

docker-compose.yml file:

services:
  app1:
    container_name: app1
    hostname: app1
    image: app1:1.0
    build: ./app1
    init: true
    entrypoint:
      - "python3"
      - "/app/app.py"
    security_opt:
      - seccomp=unconfined
    tty: true
    healthcheck:
      test: /usr/bin/wget http://127.0.0.1 || exit 1
      interval: 30s
      timeout: 60s
      retries: 5
      start_period: 300s
    restart: always
    volumes:
      - ./app1:/app
      - ./shared_data:/shared_data
    environment:
      HEALTHCHECK_DEBUG: 1
    cpu_count: 1
    cpus: 0.95
    cpu_percent: 10
    mem_limit: 25m
    networks:
      - robot_internal_net

[mheon]

I see "Custom embedded system" - Podman healthchecks require the use of systemd as an init system. Does your distribution use systemd for init?

[pclass-sensonix]

@mheon - Yes I use systemd. I can see the timer firing periodically to run the healthcheck.

[mheon]

Alright. Next best guess is that this is an issue with the Compat API for container creation - might not be translating the Docker-formatted healthchecks properly into Podman ones. @jwhonce You have time to take a look? Could also be a good one for @cdoern

[cdoern]

I can take a look at this, it is in the same vein as #10610

edit: made some progress, narrowing down where in the compat side I think the issue is. Will follow up more tomorrow when I am sure

[cdoern]

@jwhonce PTAL if you have time, not seeing any obvious issues with the compat API on this one. Healthcheck is passed in the body of the request and seems to be decoded properly.

[pclass-sensonix]

@cdoern - thanks for starting to look into this. Were you able to reproduce the failing healthchecks as I described?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@cdoern did you ever look at this?

[cdoern]

@rhatdan yes, I ccd @jwhonce because I didn't see any issues with the healthchecks. It is similar to #10610 but I don't think that fixed it. Will take another look.