Failure cluster [6140da15...] e2e Flake: Finished pod is deleted before events can be retrieved

[brianpursley]

Failure cluster 6140da152b810b30d99b

Error text:

[FAILED] timed out waiting for the condition
In [It] at: test/e2e/node/pods.go:276 @ 01/02/23 10:40:40.11

Recent failures:

1/9/2023, 3:08:19 PM pr:pull-kubernetes-e2e-kind
1/9/2023, 12:37:26 PM pr:pull-kubernetes-e2e-kind-ipv6
1/9/2023, 9:57:40 AM pr:pull-kubernetes-e2e-kind-ipv6
1/6/2023, 6:35:47 PM pr:pull-kubernetes-e2e-kind-ipv6
1/6/2023, 4:48:42 PM pr:pull-kubernetes-e2e-kind-ipv6

/kind failing-test
/kind flake

/sig node

Additional Information

It looks like the test does the following:

1. Creates a pod that has a container that immediately exits (command is /bin/true)
2. Confirms that the pod has completed successfully
3. Attempts to read the events for the pod

When this test flakes/fails, it seems like steps 1 and 2 are successful, but it times out on step 3 because apparently the pod no longer exists.

This makes me wonder if, under certain load conditions, the pod GC is cleaning up the terminated pod before step 3 (or even step 2) can do its thing.

It looks like the default value for --terminated-pod-gc-threshold is 12500,

That seems high enough that it would be unlikely that the threshold would be reached (could there really be 12500 completed pods?), but I wonder if something in the e2e tests is lowering it. I noticed this line which sets TERMINATED_POD_GC_THRESHOLD to 100, but that is gce-specific. Is it possible something similar is being set to lower the threshold for the e2e-kind tests? I did a search but didn't see anywhere that it was being set.

[aojea]

/cc @aojea @BenTheElder

interesting 👀 , we should not have tests that depends on Events , they are already documented to be best effort

The test must be changed to assert in something different, this test will be always be flaky if rely on events

[SergeyKanzhelev]

/triage accepted
/priority important-soon

[aojea]

Events are not guaranteed to be delivered

kubernetes/test/e2e/node/pods.go

Lines 276 to 292 in 2f6c4f5

	framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) {
	selector := fields.Set{
	"involvedObject.kind": "Pod",
	"involvedObject.uid": string(createdPod.UID),
	"involvedObject.namespace": f.Namespace.Name,
	"source": "kubelet",
	}.AsSelector().String()
	options := metav1.ListOptions{FieldSelector: selector}
	eventList, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).List(ctx, options)
	if err != nil {
	return false, err
	}
	if len(eventList.Items) > 0 {
	return true, nil
	}
	return false, nil
	}))

we should have another way of asserting , this is always going to be flaky

[brianpursley]

Agreed, it doesn't sound like events are meant to be used for something that requires 100% reliability.

The test in question is called should not create extra sandbox if all containers are done. Is there another way for an e2e test to know whether an extra sandbox was created?

Should this even be an e2e test, or would it be better as a unit test in kubelet (SyncPod() or computePodActions())?

There must be some reason this test was created in the first place. A regression maybe?

Without context, it doesn't seem particularly useful. It seems obvious that an extra sandbox should not be created. Is this even something we need to check? It must have happened at some point I guess.

[brianpursley]

Related Issues:

• [Flaky test] [sig-node] Pods Extended Pod Container lifecycle should not create extra sandbox if all containers are done #106904
• [Flaky test] [sig-node] Pods Extended Pod Container lifecycle should not create extra sandbox if all containers are done #109118
• An extra sandbox container created after job pod succeed #75441 (Possibly)

[k8s-triage-robot]

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged.
Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten