integration: Wait until test-pod was OOMKilled for oomkill related tests #2649
Conversation
eiffel-fl
force-pushed
the
citest/francis/integration-oomkill-wait-killed
branch
from
cca61a9
to
5bff79d
Compare
| if *k8sDistro == K8sDistroAKSUbuntu { | ||
| waitCommand = WaitUntilTestPodOOMKilledCommand(ns) | ||
| } else { | ||
| waitCommand = WaitUntilTestPodReadyCommand(ns) |
There was a problem hiding this comment.
Is the difference of behaviour really about the distribution?
If the container is slow to get oomkilled, we should be able to detect the pod be condition=ready and then oomkilled. If the container gets oomkilled fast, the pod status might go to oomkilled fast enough so we don't see the condition=ready.
It seems to me we should wait for the condition "ready or oomkilled" to avoid the race condition, if possible.
There was a problem hiding this comment.
Is the difference of behaviour really about the distribution?
So far, the tests are failing on AKS Ubuntu.
If the container is slow to get oomkilled, we should be able to detect the pod be
condition=readyand then oomkilled. If the container gets oomkilled fast, the pod status might go to oomkilled fast enough so we don't see thecondition=ready.It seems to me we should wait for the condition "ready or oomkilled" to avoid the race condition, if possible.
I am still understanding how to fix everything here.
Particularly, I am not even sure we would get event generated in this case (i.e. OOMKilled on AKS Ubuntu), so I may just end up with skipping the test here ¯\(ツ)/¯.
More specifically, the pod never reaches this state on other distribution, because it is only tail which is killed and not the pod itself.
There was a problem hiding this comment.
OK, so with using sleep I was able to have the CI passed at one time:
https://github.com/inspektor-gadget/inspektor-gadget/actions/runs/8452074988
I will try with the approach to wait for ready or OOM killed, but I do not like the idea to have to run the CI up to 6 times to have it pass.
…l tests. Signed-off-by: Francis Laniel <flaniel@linux.microsoft.com>
eiffel-fl
force-pushed
the
citest/francis/integration-oomkill-wait-killed
branch
from
5bff79d
to
3ff5a59
Compare
eiffel-fl
force-pushed
the
citest/francis/integration-oomkill-wait-killed
branch
from
080170c
to
3ff5a59
Compare
|
I will merge it as is. |
|
Thank you for the review! |
No description provided.