kubelet: fix create sandbox delete pod race

[rphillips]

What type of PR is this?
/kind bug
/sig node
/kind flake

What this PR does / why we need it:
This PR addresses a race condition where createPodSandbox can return an error from various external modules (CNI, CRI, CSI) when the pod has been deleted. This is not an error case, so we should not log or record an event. I have created a log message at log level 4 if anyone would like to trace this code path.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

[ehashman]

/triage accepted
/priority important-soon
/lgtm

[rphillips]

/retest

[rphillips]

rebased with current master

[ehashman]

I wonder if we've broken the assumptions here, causing this test to fail:

kubernetes/test/e2e_node/mirror_pod_grace_period_test.go

Lines 67 to 78 in 838bb6a

	for {
	if time.Now().Sub(start).Seconds() > 19 {
	break
	}
	pod, err := f.ClientSet.CoreV1().Pods(ns).Get(context.TODO(), mirrorPodName, metav1.GetOptions{})
	framework.ExpectNoError(err)
	if pod.Status.Phase != v1.PodRunning {
	framework.Failf("expected the mirror pod %q to be running, got %q", mirrorPodName, pod.Status.Phase)
	}
	// have some pause in between the API server queries to avoid throttling
	time.Sleep(time.Duration(200) * time.Millisecond)
	}

The mirror pod has been deleted by the time we hit that code block, so I'm not sure why the test is asserting it will still be there.

The other failures look like possible flakes,
/retest

[ehashman]

👍

[rphillips]

/retest

[rphillips]

... pushed the wrong button on the retest

[rphillips]

/retest

[ehashman]

/lgtm

[mrunalp]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ehashman, mrunalp, rphillips

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/kubelet/OWNERS [mrunalp]
• staging/src/k8s.io/cri-api/pkg/OWNERS [mrunalp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[rphillips]

/retest

[rphillips]

/retest

[rphillips]

/retest

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.