Flake in node e2e: "Container Runtime Conformance Test container conformance blackbox test when running a container that terminates [It] it should be possible to delete [Conformance] "

[smarterclayton]

Summarizing 1 Failure:

[Fail] Container Runtime Conformance Test container conformance blackbox test when running a container that terminates [It] it should be possible to delete [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:76

Ran 22 of 22 Specs in 1205.295 seconds

[mikedanese]

Again here:

http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubelet-gce-e2e-ci/2233

07:46:01 • Failure [0.085 seconds]
07:46:01 Kubelet Container Manager
07:46:01 /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:107
07:46:01   oom score adjusting
07:46:01   /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:105
07:46:01     when scheduling a busybox command that always fails in a pod
07:46:01     /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:104
07:46:01       it should be possible to delete [It]
07:46:01       /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:91
07:46:01 
07:46:01       Error deleting Pod pods "bin-false" not found
07:46:01       Expected
07:46:01           <*errors.StatusError | 0xc82017dc80>: {
07:46:01               ErrStatus: {
07:46:01                   TypeMeta: {Kind: "Status", APIVersion: "v1"},
07:46:01                   ListMeta: {SelfLink: "", ResourceVersion: ""},
07:46:01                   Status: "Failure",
07:46:01                   Message: "pods \"bin-false\" not found",
07:46:01                   Reason: "NotFound",
07:46:01                   Details: {Name: "bin-false", Group: "", Kind: "pods", Causes: nil, RetryAfterSeconds: 0},
07:46:01                   Code: 404,
07:46:01               },
07:46:01           }
07:46:01       to be nil
07:46:01 
07:46:01       /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:90

cc @pwittrock

[Random-Liu]

@liangchenye

[liangchenye]

Not sure how does this error come out. The 'Delete' test case only does two things: create a 'bin-false' pod and delete it immediately after that. Since there is no error comes out in pod creation, it means before the 'Delete' test case, there is no 'bin-false' pod (neither never been created nor has already been deleted). But this issue shows, after the creation, it is removed by another test case ('Terminate Reason' test case).
So, I guess it should be something like this: in the 'Delete' test, it creates a pod at B point
and delete the pod at D point. I made a test, if it creates a pod at A point, there will be a creation error.

'Terminate Reason' test case: 
     create -- check-- delete --  waiting --- real deleted 
                                 |         |                 |
'Delete' test case:              A         B                  D

So is this a bug of k8s apiserver? The 'real deleted' status seems not synchronized.

Anyway I'll choose different name in different cases in the later development. (and different namespace in different test file)

[jsafrane]

This looks similar to the first comment, different to @mikedanese's one: https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/24861/node-pull-build-e2e-test/3451/build-log.txt

 Failure [0.046 seconds]
Container Runtime Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:139
  container conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:138
    when running a container that terminates
    /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:91
      it should be possible to delete [Conformance] [It]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:81

      Expected error:
          <*errors.StatusError | 0xc8205d4000>: {
              ErrStatus: {
                  TypeMeta: {Kind: "Status", APIVersion: "v1"},
                  ListMeta: {SelfLink: "", ResourceVersion: ""},
                  Status: "Failure",
                  Message: "pods \"busybox\" not found",
                  Reason: "NotFound",
                  Details: {Name: "busybox", Group: "", Kind: "pods", Causes: nil, RetryAfterSeconds: 0},
                  Code: 404,
              },
          }
          pods "busybox" not found
      not to have occurred

[liangchenye]

will be fixed by #24191