Use persist dir for oom file

[umohnani8]

Conmon writes the exit file and oom file (if container
was oom killed) to the persist directory. This directory
is retained across reboots as well.
Update podman to create a persist-dir/ctr-id for the exit
and oom files for each container to be written to. The oom
state of container is set after reading the files
from the persist-dir/ctr-id directory.
The exit code still continues to read the exit file from
the exits directory.

Fixes #13102

Does this PR introduce a user-facing change?

Use the conmon persist-dir for reading the container's oom file if oom killed.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: umohnani8

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [umohnani8]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[packit-as-a-service]

Cockpit tests failed for commit 67b526e. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit 0c224be. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit d3c5dab. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit c7e8ec1. @martinpitt, @jelly, @mvollmer please check.

[packit-as-a-service]

Cockpit tests failed for commit c2d7ab8. @martinpitt, @jelly, @mvollmer please check.

[umohnani8]

@rhatdan @mheon @haircommander PTAL

This is a breaking change as we are moving from using exit-dir to perisist-dir for the exit and oom files. Conmon writes the oom file to the persist-dir if a container is oom killed as well as the exit file so switching to only using persist-dir.

This fixes the bug also where we were not setting oom killed to true when the container was oom killed.

[haircommander]

this makes sense to me. exit dir is most useful for cri-o which registers an inotify watcher on it. Having the exists centralized is helpful. Whereas podman really only needs to read from one location, and having that be persist path makes sense

[mheon]

LGTM

[rhatdan]

/lgtm
/hold

[umohnani8]

Is the hyperv failure expected?

[mheon]

No, but it could be a flake

[Luap99]

Will this break all existing running containers? They can no longer read the exit file correctly.
I would love to have #21535 running on this before merging. If this doesn't work then we cannot really upgrade test properly.

[umohnani8]

Sure we can wait on #21535 to get in before getting this in.

And yes it is a breaking change, any existing containers will need to be re-created.
This is no longer a breaking change.

[packit-as-a-service]

Ephemeral COPR build failed. @containers/packit-build please check.

[umohnani8]

So this is no longer a breaking change as we only use the persist dir for the oom file now. We were running into some timeout issues when switching to persist dir completely for the exit files and I think that was because we also use an inotify watcher.

@mheon @rhatdan @Luap99 this is ready, PTAL.

[Luap99]

code LGTM but given that touches exec parts as well I guess we should test an podman exec oom kill as well here. I am not sure what happens if podman exec runs into an oom kill than does it a) only kill the exec session or b) the whole container or c ) can it be either depending on what the oom killer feels like or something else entirely?

[haircommander]

a) only kill the exec session or b) the whole container or c ) can it be either depending on what the oom killer feels like or something else entirely?

should be the exec session unless podman is setting the cgroupv2 knob memory.oom.group to 1 for the container's cgroup

though, with v1 oom killing, sometimes it's c :)

[umohnani8]

Nothing with the exec functionality has been changed by adding the persist dir option here. The only thing is if an exec session is oom killed, I believe it will write oom file in the persist dir location and handle the exit code as it always was.

I am trying to add a test regardless, which works as expected when I do it manually but not in the gingko test suite. These are the commands I am running

podman run -d --name memory-test --memory 3M --oom-score-adj 1000 alpine tail -f /dev/null
podman exec -it memory-test sh -c "dd if=/dev/zero of=/dev/null bs=10M"

Any other possible examples that would work with the test suite?

[rhatdan]

/unhold

If something has to be done for exec lets do it in a separate pr.

[rhatdan]

/lgtm