Cannot determine if a container was unable to start

[paralin]

/kind feature

Description

Say I execute a pod with the following pod spec:

        restartPolicy: OnFailure
        containers:
        - image: docker.io/library/alpine:edge
          name: hello
          args:
          - efcho
          - Hello world
          tty: true

The resulting container, of course, cannot start:

podman start container-id
Error: unable to start container "container-id": crun: executable file `efcho` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found

... due to command not found.

However, this error does not appear anywhere in the "podman inspect" output:

[
     {
          "State": {
               "OciVersion": "1.0.2-dev",
               "Status": "created",
               "Running": false,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 0,
               "ExitCode": 0,
               "Error": "",
               "StartedAt": "0001-01-01T00:00:00Z",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "Health": {
                    "Status": "",
                    "FailingStreak": 0,
                    "Log": null
               },
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          }
     }
]

... the container just appears as "created" and there's no way to distinguish why it failed.

Describe the results you received:

What is the correct way to check if a container is in this kind of failed state via the API, and how to get the error message?

Is the best way to just check if the container is in "created" state, try to start it via api, and check for an error?

Describe the results you expected:

This seems like a bit of a hack/workaround, it'd be best if the error was in the container status somewhere.

[paralin]

For reference, this is what I'm doing now as a workaround:

https://gist.github.com/paralin/ad4279d14a588eb3e897519b9f299907

... the error is only exposed when calling Start on the container, so if the Pod fails to start (created state), I have to loop over the containers & call Start on each until one returns an error.

[mheon]

So, to be perfectly clear, the request is to include the error in the Error field of container inspect's State struct?

[paralin]

@mheon That would work, that & also hopefully some indication of the exit code or the nature of the failure.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[paralin]

This is still relevant, I think, there's no non hacky way to determine if a container failed to start

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon any chance you can make this happen? Or should we hand this to an intern?

[mheon]

Sure. Just need to add a field to the DB to store the error message from the last run, and throw that in podman inspect.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Any chance you can implement this?

[mheon]

Sure, I'll put it on the list

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Any progress on this?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@mheon Reminder.

[mheon]

On it

[jakecorrenti]

@mheon if you're not too far along with this I wouldn't mind taking a stab at it

[mheon]

@jakecorrenti Feel free to take a stab at it!

[jakecorrenti]

I currently have a partial fix for this, but it isn't capturing the entire scope of the issue. In

podman/libpod/container_api.go

Line 85 in 639efd8

func (c *Container) Start(ctx context.Context, recursive bool) error {

I currently have something along these lines

saveErrorState := func(e error) error {
    c.state.Error = e.Error()
    if err := c.save(); err != nil {
        return err
    }
    return nil
}

which is called when an error occurs in the function. The issue is this doesn't allow for the entire issue to be solved. If an error occurs in the container engine's ContainerRun function, I don't have the availability to modify the container's state and save it (to the best of my knowledge) without modifying the API in some way. Is there a way to do this without making any changes to the API?

For reference, this is the result of the above fix in regards to the issue above

         "State": {
              "OciVersion": "1.0.2-dev",
              "Status": "created",
              "Running": false,
              "Paused": false,
              "Restarting": false,
              "OOMKilled": false,
              "Dead": false,
              "Pid": 0,
              "ExitCode": 0,
              "Error": "crun: executable file `efcho` not found in $PATH: No such file or directory: OCI runtime attempted to invoke a command that was not found",
              "StartedAt": "0001-01-01T00:00:00Z",
              "FinishedAt": "0001-01-01T00:00:00Z",
              "Health": {
                   "Status": "",
                   "FailingStreak": 0,
                   "Log": null
              },
              "CheckpointedAt": "0001-01-01T00:00:00Z",
              "RestoredAt": "0001-01-01T00:00:00Z"
         },

[mheon]

Honestly, I think it's sufficient to just capture the Libpod bits... Probably would even be sufficient to capture just errors out of the OCI runtime, even.

[jakecorrenti]

sounds good