Skip to content
This repository has been archived by the owner on May 12, 2021. It is now read-only.

k8s replication controller not working #1931

Closed
GabyCT opened this issue Aug 6, 2019 · 4 comments
Closed

k8s replication controller not working #1931

GabyCT opened this issue Aug 6, 2019 · 4 comments
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
Issue backlog

Comments

@GabyCT
Copy link
Contributor

GabyCT commented Aug 6, 2019

While running the replication k8s controller test, we have that the pod is never created

$ kubectl describe pods
  Normal   Scheduled               43s               default-scheduler    Successfully assigned default/replicationtest-gf5pz to gabycentos
  Warning  FailedCreatePodSandBox  3s (x4 over 43s)  kubelet, gabycentos  Failed create pod sandbox: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]

$ kubectl get pods
NAME                    READY   STATUS              RESTARTS   AGE
replicationtest-gf5pz   0/1     ContainerCreating   0          82s

This is the logs

ago 06 00:58:55 gabycentos kubelet[46027]: E0806 00:58:55.022953   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:58:55 gabycentos kubelet[46027]: E0806 00:58:55.024103   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:58:55 gabycentos kubelet[46027]: E0806 00:58:55.024173   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:58:55 gabycentos kubelet[46027]: E0806 00:58:55.024251   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 00:59:08 gabycentos kubelet[46027]: E0806 00:59:08.011716   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:08 gabycentos kubelet[46027]: E0806 00:59:08.011827   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:08 gabycentos kubelet[46027]: E0806 00:59:08.011849   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:08 gabycentos kubelet[46027]: E0806 00:59:08.011898   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 00:59:20 gabycentos kubelet[46027]: E0806 00:59:20.010758   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:20 gabycentos kubelet[46027]: E0806 00:59:20.010842   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:20 gabycentos kubelet[46027]: E0806 00:59:20.010863   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:20 gabycentos kubelet[46027]: E0806 00:59:20.010942   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 00:59:31 gabycentos kubelet[46027]: E0806 00:59:31.013220   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:31 gabycentos kubelet[46027]: E0806 00:59:31.013300   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:31 gabycentos kubelet[46027]: E0806 00:59:31.013348   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:31 gabycentos kubelet[46027]: E0806 00:59:31.013436   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 00:59:44 gabycentos kubelet[46027]: E0806 00:59:44.011659   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:44 gabycentos kubelet[46027]: E0806 00:59:44.011724   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:44 gabycentos kubelet[46027]: E0806 00:59:44.011741   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:44 gabycentos kubelet[46027]: E0806 00:59:44.011807   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 00:59:57 gabycentos kubelet[46027]: E0806 00:59:57.216909   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:57 gabycentos kubelet[46027]: E0806 00:59:57.216998   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:57 gabycentos kubelet[46027]: E0806 00:59:57.217018   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 00:59:57 gabycentos kubelet[46027]: E0806 00:59:57.217188   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:00:09 gabycentos kubelet[46027]: E0806 01:00:09.014869   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:09 gabycentos kubelet[46027]: E0806 01:00:09.015538   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:09 gabycentos kubelet[46027]: E0806 01:00:09.015578   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:09 gabycentos kubelet[46027]: E0806 01:00:09.015684   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:00:22 gabycentos kubelet[46027]: E0806 01:00:22.014803   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:22 gabycentos kubelet[46027]: E0806 01:00:22.014889   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:22 gabycentos kubelet[46027]: E0806 01:00:22.014919   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:22 gabycentos kubelet[46027]: E0806 01:00:22.015000   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:00:38 gabycentos kubelet[46027]: E0806 01:00:38.015764   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:38 gabycentos kubelet[46027]: E0806 01:00:38.015835   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:38 gabycentos kubelet[46027]: E0806 01:00:38.015872   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:38 gabycentos kubelet[46027]: E0806 01:00:38.015931   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:00:50 gabycentos kubelet[46027]: E0806 01:00:50.013298   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:50 gabycentos kubelet[46027]: E0806 01:00:50.013367   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:50 gabycentos kubelet[46027]: E0806 01:00:50.013395   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:00:50 gabycentos kubelet[46027]: E0806 01:00:50.013479   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:01:02 gabycentos kubelet[46027]: E0806 01:01:02.046519   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:02 gabycentos kubelet[46027]: E0806 01:01:02.046583   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:02 gabycentos kubelet[46027]: E0806 01:01:02.046614   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:02 gabycentos kubelet[46027]: E0806 01:01:02.046741   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:01:14 gabycentos kubelet[46027]: E0806 01:01:14.013618   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:14 gabycentos kubelet[46027]: E0806 01:01:14.013695   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:14 gabycentos kubelet[46027]: E0806 01:01:14.013716   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:14 gabycentos kubelet[46027]: E0806 01:01:14.013804   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
ago 06 01:01:29 gabycentos kubelet[46027]: E0806 01:01:29.012481   46027 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:29 gabycentos kubelet[46027]: E0806 01:01:29.012545   46027 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:29 gabycentos kubelet[46027]: E0806 01:01:29.012564   46027 kuberuntime_manager.go:692] createPodSandbox for pod "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]
ago 06 01:01:29 gabycentos kubelet[46027]: E0806 01:01:29.012631   46027 pod_workers.go:190] Error syncing pod 0308105f-e70a-4564-a678-2cfcd1d91cb1 ("replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)"), skipping: failed to "CreatePodSandbox" for "replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)" with CreatePodSandboxError: "CreatePodSandbox for pod \"replicationtest-gf5pz_default(0308105f-e70a-4564-a678-2cfcd1d91cb1)\" failed: rpc error: code = Unknown desc = failed to find runtime handler kata from runtime list map[runc:0xc00001da70]"
@GabyCT GabyCT added bug Incorrect behaviour needs-review Needs to be assessed by the team. labels Aug 6, 2019
GabyCT added a commit to GabyCT/tests-1 that referenced this issue Aug 6, 2019
@GabyCT
test: Skip k8s replication test
f132752 
It seems that there is an issue while running the replication test
kata-containers/runtime#1931 so we need to
skip the test.

Fixes kata-containers#1900

Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
@GabyCT GabyCT mentioned this issue Aug 6, 2019
Merged
@grahamwhaley
Copy link
Contributor

Most odd, it looks like it is failing to find runc:

...failed to find runtime handler kata from runtime list map[runc:0xc00001da70]

Unclear if that was via the 'default' runtime or if we set RuntimeClass to runc (which afaik does not work, as nobody registers runc as a runtimeclass item... ).

@chavafg
Copy link
Contributor

chavafg commented Aug 6, 2019

@GabyCT can you share your runtime list map from the crio.conf file and check if the crio log shows more info?

Also, I was wondering if you were able to launch other pods?

@GabyCT
Copy link
Contributor Author

GabyCT commented Aug 6, 2019

@chavafg thanks for the feedback so it seems that I was able to launch other tests. @ganeshmaharaj , locally I can reproduce the issue with centos 7.5

ok 1 Replication controller
1..1
ok 1 Replication controller
1..1
not ok 1 Replication controller
# (from function `waitForProcess' in file ../../.ci/lib.sh, line 165,
#  in test file k8s-replication.bats, line 42)
#   `waitForProcess "$wait_time" "$sleep_time" "$cmd"' failed
# INFO: k8s configured to use runtimeclass
# replicationcontroller/replicationtest created
#   Normal  SuccessfulCreate  0s    replication-controller  Created pod: replicationtest-zwqwp
# pod/replicationtest-zwqwp condition met
# timed out waiting for the condition on pods/replicationtest-mncqb
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# pod/replicationtest-zwqwp condition met
# Error from server (NotFound): pods "replicationtest-mncqb" not found
# Error from server (NotFound): pods "replicationtest-t6bzd" not found
# Error from server (NotFound): pods "replicationtest-mncqb replicationtest-t6bzd replicationtest-zwqwp" not found

I put the test on a loop and fails like the CI

@GabyCT
Copy link
Contributor Author

GabyCT commented Aug 6, 2019

This is the log

ago 06 18:56:01 gabycentos kubelet[126556]: I0806 18:56:01.229158  126556 reconciler.go:203] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-q8rdh" (UniqueName: "kubernetes.io/secret/eab7b2d0-dd1f-47ca-b4c8-cd622fedf6d5-default-token-q8rdh") pod "replicationtest-jsphd" (UID: "eab7b2d0-dd1f-47ca-b4c8-cd622fedf6d5")
ago 06 18:56:02 gabycentos kubelet[126556]: E0806 18:56:02.248059  126556 kubelet_volumes.go:154] Orphaned pod "cf55ca28-776f-48db-87ce-3144b648079d" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
ago 06 18:56:03 gabycentos kubelet[126556]: E0806 18:56:03.588023  126556 remote_runtime.go:295] ContainerStatus "9f406ded73ba9dc3c482cb42228ed9b4585ffd318cdb49635cd4dbc737a2d0eb" from runtime service failed: rpc error: code = Unknown desc = container with ID starting with 9f406ded73ba9dc3c482cb42228ed9b4585ffd318cdb49635cd4dbc737a2d0eb not found: ID does not exist
ago 06 18:56:04 gabycentos kubelet[126556]: E0806 18:56:04.247420  126556 kubelet_volumes.go:154] Orphaned pod "cf55ca28-776f-48db-87ce-3144b648079d" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.
ago 06 18:56:04 gabycentos kubelet[126556]: I0806 18:56:04.545352  126556 reconciler.go:177] operationExecutor.UnmountVolume started for volume "default-token-q8rdh" (UniqueName: "kubernetes.io/secret/6442872c-8d40-45c2-9f9c-6fde4961aa57-default-token-q8rdh") pod "6442872c-8d40-45c2-9f9c-6fde4961aa57" (UID: "6442872c-8d40-45c2-9f9c-6fde4961aa57")
ago 06 18:56:04 gabycentos kubelet[126556]: I0806 18:56:04.556997  126556 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6442872c-8d40-45c2-9f9c-6fde4961aa57-default-token-q8rdh" (OuterVolumeSpecName: "default-token-q8rdh") pod "6442872c-8d40-45c2-9f9c-6fde4961aa57" (UID: "6442872c-8d40-45c2-9f9c-6fde4961aa57"). InnerVolumeSpecName "default-token-q8rdh". PluginName "kubernetes.io/secret", VolumeGidValue ""
ago 06 18:56:04 gabycentos kubelet[126556]: I0806 18:56:04.645691  126556 reconciler.go:297] Volume detached for volume "default-token-q8rdh" (UniqueName: "kubernetes.io/secret/6442872c-8d40-45c2-9f9c-6fde4961aa57-default-token-q8rdh") on node "gabycentos" DevicePath ""
ago 06 18:56:06 gabycentos kubelet[126556]: E0806 18:56:06.246877  126556 kubelet_volumes.go:154] Orphaned pod "cf55ca28-776f-48db-87ce-3144b648079d" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.

@jodh-intel jodh-intel added this to To do in Issue backlog Aug 10, 2020
egernst pushed a commit to egernst/runtime that referenced this issue Feb 9, 2021
@bergwolf
vendor: update grpc-go dependency
cd03e0c 
This updates grpc-go vendor package to v1.11.3 release, to fix server.Stop()
handling so that server.Serve() does not wait blindly.

Full commit list:
d11072e (tag: v1.11.3) Change version to 1.11.3
d06e756 clientconn: add support for unix network in DialContext. (kata-containers#1883)
452c2a7 Change version to 1.11.3-dev
d89cded (tag: v1.11.2) Change version to 1.11.2
98ac976 server: add grpc.Method function for extracting method from context (kata-containers#1961)
0f5fa28 Change version to 1.11.2-dev
1e2570b (tag: v1.11.1) Change version to 1.11.1
d28faca client: Fix race when using both client-side default CallOptions and per-call CallOptions (kata-containers#1948)
48b7669 Change version to 1.11.1-dev
afc05b9 (tag: v1.11.0) Change version to 1.11.0
f2620c3 resolver: keep full unparsed target string if scheme in parsed target is not registered (kata-containers#1943)
9d2250f status: rename Status to GRPCStatus to avoid name conflicts (kata-containers#1944)
2756956 status: Allow external packages to produce status-compatible errors (kata-containers#1927)
0ff1b76 routeguide: reimplement distance calculation
dfbefc6 service reflection can lookup enum, enum val, oneof, and field symbols (kata-containers#1910)
32d9ffa Documentation: Fix broken link in rpc-errors.md (kata-containers#1935)
d5126f9 Correct Go 1.6 support policy (kata-containers#1934)
5415d18 Add documentation and example of adding details to errors (kata-containers#1915)
57640c0 Allow storing alternate transport.ServerStream implementations in context (kata-containers#1904)
031ee13 Fix Test: Update the deadline since small deadlines are prone to flakes on Travis. (kata-containers#1932)
2249df6 gzip: Add ability to set compression level (kata-containers#1891)
8124abf credentials/alts: Remove the enable_untrusted_alts flag (kata-containers#1931)
b96718f metadata: Fix bug where AppendToOutgoingContext could modify another context's metadata (kata-containers#1930)
738eb6b fix minor typos and remove grpc.Codec related code in TestInterceptorCanAccessCallOptions (kata-containers#1929)
211a7b7 credentials/alts: Update ALTS "New" APIs (kata-containers#1921)
fa28bef client: export types implementing CallOptions for access by interceptors (kata-containers#1902)
ec9275b travis: add Go 1.10 and run vet there instead of 1.9 (kata-containers#1913)
13975c0 stream: split per-attempt data from clientStream (kata-containers#1900)
2c2d834 stats: add BeginTime to stats.End (kata-containers#1907)
3a9e1ba Reset ping strike counter right before sending out data. (kata-containers#1905)
90dca43 resolver: always fall back to default resolver when target does not follow URI scheme (kata-containers#1889)
9aba044 server: Convert all non-status errors to codes.Unknown (kata-containers#1881)
efcc755 credentials/alts: change ALTS protos to match the golden version (kata-containers#1908)
0843fd0 credentials/alts: fix infinite recursion bug [in custom error type] (kata-containers#1906)
207e276 Fix test race: Atomically access minConnecTimout in testing environment. (kata-containers#1897)
3ae2a61 interop: Add use_alts flag to client and server binaries (kata-containers#1896)
5190b06 ALTS: Simplify "New" APIs (kata-containers#1895)
7c5299d Fix flaky test: TestCloseConnectionWhenServerPrefaceNotReceived (kata-containers#1870)
f0a1202 examples: Replace context.Background with context.WithTimeout (kata-containers#1877)
a1de3b2 alts: Change ALTS proto package name (kata-containers#1886)
2e7e633 Add ALTS code (kata-containers#1865)
583a630 Expunge error codes that shouldn't be returned from library (kata-containers#1875)
2759199 Small spelling fixes (unknow -> unknown) (kata-containers#1868)
12da026 clientconn: fix a typo in GetMethodConfig documentation (kata-containers#1867)
dfa1834 Change version to 1.11.0-dev (kata-containers#1863)
46fd263 benchmarks: add flag to benchmain to use bufconn instead of network (kata-containers#1837)
3926816 addrConn: Report underlying connection error in RPC error (kata-containers#1855)
445b728 Fix data race in TestServerGoAwayPendingRPC (kata-containers#1862)
e014063 addrConn: keep retrying even on non-temporary errors (kata-containers#1856)
484b3eb transport: fix race causing flow control discrepancy when sending messages over server limit (kata-containers#1859)
6c48c7f interop test: Expect io.EOF from stream.Send() (kata-containers#1858)
08d6261 metadata: provide AppendToOutgoingContext interface (kata-containers#1794)
d50734d Add status.Convert convenience function (kata-containers#1848)
365770f streams: Stop cleaning up after orphaned streams (kata-containers#1854)
7646b53 transport: support stats.Handler in serverHandlerTransport (kata-containers#1840)
104054a Fix connection drain error message (kata-containers#1844)
d09ec43 Implement unary functionality using streams (kata-containers#1835)
37346e3 Revert "Add WithResolverUserOptions for custom resolver build options" (kata-containers#1839)
424e3e9 Stream: do not cancel ctx created with service config timeout (kata-containers#1838)
f9628db Fix lint error and typo (kata-containers#1843)
0bd008f stats: Fix bug causing trailers-only responses to be reported as headers (kata-containers#1817)
5769e02 transport: remove unnecessary rstReceived (kata-containers#1834)
0848a09 transport: remove redundant check of stream state in Write (kata-containers#1833)
c22018a client: send RST_STREAM on client-side errors to prevent server from blocking (kata-containers#1823)
82e9f61 Use keyed fields for struct initializers (kata-containers#1829)
5ba054b encoding: Introduce new method for registering and choosing codecs (kata-containers#1813)
4f7a2c7 compare atomic and mutex performance in case of contention. (kata-containers#1788)
b71aced transport: Fix a data race when headers are received while the stream is being closed (kata-containers#1814)
46bef23 Write should fail when the stream was done but context wasn't cancelled. (kata-containers#1792)
10598f3 Explain target format in DialContext's documentation (kata-containers#1785)
08b7bd3 gzip: add Name const to avoid typos in usage (kata-containers#1804)
8b02d69 remove .please-update (kata-containers#1800)
1cd2346 Documentation: update broken wire.html link in metadata package. (kata-containers#1791)
6913ad5 Document that all errors from RPCs are status errors (kata-containers#1782)
8a8ac82 update const order (kata-containers#1770)
e975017 Don't set reconnect parameters when the server has already responded. (kata-containers#1779)
7aea499 credentials: return Unavailable instead of Internal for per-RPC creds errors (kata-containers#1776)
c998149 Avoid copying headers/trailers in unary RPCs unless requested by CallOptions (kata-containers#1775)
8246210 Update version to 1.10.0-dev (kata-containers#1777)
17c6e90 compare atomic and mutex performance for incrementing/storing one variable (kata-containers#1757)
65c901e Fix flakey test. (kata-containers#1771)
7f2472b grpclb: Remove duplicate init() (kata-containers#1764)
09fc336 server: fix bug preventing Serve from exiting when Listener is closed (kata-containers#1765)
035eb47 Fix TestGracefulStop flakiness (kata-containers#1767)
2720857 server: fix race between GracefulStop and new incoming connections (kata-containers#1745)
0547980 Notify parent ClientConn to re-resolve in grpclb (kata-containers#1699)
e6549e6 Add dial option to set balancer (kata-containers#1697)
6610f9a Fix test: Data race while resetting global var. (kata-containers#1748)
f4b5237 status: add Code convenience function (kata-containers#1754)
47bddd7 vet: run golint on _string files (kata-containers#1749)
45088c2 examples: fix concurrent map accesses in route_guide server (kata-containers#1752)
4e393e0 grpc: fix deprecation comments to conform to standard (kata-containers#1691)
0b24825 Adjust keepalive paramenters in the test such that scheduling delays don't cause false failures too often. (kata-containers#1730)
f9390a7 fix typo (kata-containers#1746)
6ef45d3 fix stats flaky test (kata-containers#1740)
98b17f2 relocate check for shutdown in ac.tearDown() (kata-containers#1723)
5ff10c3 fix flaky TestPickfirstOneAddressRemoval (kata-containers#1731)
2625f03 bufconn: allow readers to receive data after writers close (kata-containers#1739)
b0e0950 After sending second goaway close conn if idle. (kata-containers#1736)
b8cf13e Make sure all goroutines have ended before restoring global vars. (kata-containers#1732)
4742c42 client: fix race between server response and stream context cancellation (kata-containers#1729)
8fba5fc In gracefull stop close server transport only after flushing status of the last stream. (kata-containers#1734)
d1fc8fa Deflake tests that rely on Stop() then Dial() not reconnecting (kata-containers#1728)
dba60db Switch balancer to grpclb when at least one address is grpclb address (kata-containers#1692)
ca1b23b Update CONTRIBUTING.md to CNCF CLA
2941ee1 codes: Add UnmarshalJSON support to Code type (kata-containers#1720)
ec61302 naming: Fix build constraints for go1.6 and go1.7 (kata-containers#1718)
b8191e5 remove stringer and go generate (kata-containers#1715)
ff1be3f Add WithResolverUserOptions for custom resolver build options (kata-containers#1711)
580defa Fix grpc basics link in route_guide example (kata-containers#1713)
b7dc71e Optimize codes.String() method using a switch instead of a slice of indexes (kata-containers#1712)
1fc873d Disable ccBalancerWrapper when it is closed (kata-containers#1698)
bf35f1b Refactor roundrobin to support custom picker (kata-containers#1707)
4308342 Change parseTimeout to not handle non-second durations (kata-containers#1706)
be07790 make load balancing policy name string case-insensitive (kata-containers#1708)
cd563b8 protoCodec: avoid buffer allocations if proto.Marshaler/Unmarshaler (kata-containers#1689)
61c6740 Add comments to ClientConn/SubConn interfaces to indicate new methods may be added (kata-containers#1680)
ddbb27e client: backoff before reconnecting if an HTTP2 server preface was not received (kata-containers#1648)
a4bf341 use the request context with net/http handler (kata-containers#1696)
c6b4608 transport: fix race sending RPC status that could lead to a panic (kata-containers#1687)
00383af Fix misleading default resolver scheme comments (kata-containers#1703)
a62701e Eliminate data race in ccBalancerWrapper (kata-containers#1688)
1e1a47f Re-resolve target when one connection becomes TransientFailure (kata-containers#1679)
2ef021f New grpclb implementation (kata-containers#1558)
10873b3 Fix panics on balancer and resolver updates (kata-containers#1684)
646f701 Change version to 1.9.0-dev (kata-containers#1682)

Fixes: kata-containers#307

Signed-off-by: Peng Tao <bergwolf@gmail.com>
Issue backlog automation moved this from To do to Done Apr 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug Incorrect behaviour needs-review Needs to be assessed by the team.
Projects
Issue backlog
  
Done
Milestone
No milestone
Development

No branches or pull requests
4 participants
@grahamwhaley @chavafg @jodh-intel @GabyCT