Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite} #33341

Closed
k8s-github-robot opened this issue Sep 22, 2016 · 15 comments
Closed

[k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite} #33341

k8s-github-robot opened this issue Sep 22, 2016 · 15 comments
Assignees
@vishh @dchen1107
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Milestone
v1.6

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/16088/

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Expected error:
    <*errors.errorString | 0xc820f6cae0>: {
        s: "Only 1 pods started out of 2",
    }
    Only 1 pods started out of 2
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:345

Previous issues for this test: #27196 #28998 #32403
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. labels Sep 22, 2016
This was referenced Sep 27, 2016
Closed
Closed
Closed
Closed
Closed
This was referenced Oct 4, 2016
Closed
Closed
@k8s-github-robot
Copy link
Author

k8s-github-robot commented Oct 11, 2016 •
edited by rmmh

Builds:
kubernetes-e2e-gci-gke-release-1.4 1152
kubernetes-e2e-gci-gke 1643
kubernetes-e2e-gke 18164 19004

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Expected error:
    <*errors.errorString | 0xc820d1b460>: {
        s: "Only 0 pods started out of 1",
    }
    Only 0 pods started out of 1
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:393

This was referenced Oct 15, 2016
Closed
Closed
Closed
@jszczepkowski
Copy link
Contributor

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/18164/

According to build log, pod rc-light-x68aj was scheduled at 06:13:49. However, kubelet was not able to run it for 2 minutes. I attach kubelets log related to the pod. @vishh @dchen1107 can you please explain what is going on with the pod (or assign it to someone else from kubelet team for help)?

build logs:

[1mSTEP�[0m: creating replication controller rc-light in namespace e2e-tests-horizontal-pod-autoscaling-gtjuk
I1020 06:13:39.726781    2701 runners.go:100] Created replication controller with name: rc-light, namespace: e2e-tests-horizontal-pod-autoscaling-gtjuk, replica count: 2
I1020 06:13:49.727146    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:13:59.727387    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:09.727654    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:19.727999    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:29.728374    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:39.728702    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:49.728996    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:14:59.729319    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:09.729647    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:19.729984    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:29.730228    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:39.730504    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:49.730770    2701 runners.go:100] rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1020 06:15:49.758888    2701 runners.go:100] Pod rc-light-4os1u    gke-jenkins-e2e-default-pool-950b2b00-o82g  Running <nil>
I1020 06:15:49.759039    2701 runners.go:100] Pod rc-light-x68aj    gke-jenkins-e2e-default-pool-950b2b00-zonr  Pending <nil>

kubelet-logs:

I1020 13:13:39.777501    3843 kubelet.go:1592] SyncLoop (ADD, "api"): "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)"
...
I1020 13:13:40.678450    3843 docker_manager.go:1941] Need to restart pod infra container for "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)" because it is not found
E1020 13:13:40.750364    3843 docker_manager.go:761] Logging security options: {key:seccomp value:unconfined msg:}
I1020 13:13:40.813389    3843 kubelet.go:1608] SyncLoop (DELETE, "api"): "test-pod_e2e-tests-e2e-kubelet-etc-hosts-euv2e(dcf6ac25-96c6-11e6-92eb-42010af00012)"
I1020 13:13:40.819000    3843 kubelet.go:1602] SyncLoop (REMOVE, "api"): "test-pod_e2e-tests-e2e-kubelet-etc-hosts-euv2e(dcf6ac25-96c6-11e6-92eb-42010af00012)"
I1020 13:13:40.819261    3843 kubelet.go:1777] Failed to delete pod "test-pod_e2e-tests-e2e-kubelet-etc-hosts-euv2e(dcf6ac25-96c6-11e6-92eb-42010af00012)", err: pod not found
I1020 13:13:41.138484    3843 docker_manager.go:2202] Determined pod ip after infra change: "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)": "10.180.1.56"
I1020 13:13:41.263758    3843 kubelet.go:1627] SyncLoop (PLEG): "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)", event: &pleg.PodLifecycleEvent{ID:"0465c18a-96c7-11e6-92eb-42010af00012", Type:"ContainerStarted", Data:"7031755f9d909e4a0b87a955b5e0b61ea30f562e8c14ce286e5ac5d62bfae420"}
...
I1020 13:13:50.598533    3843 docker_manager.go:754] Container "rc-light" of pod "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)" created with warnings: [Your kernel does not support CPU cfs period. Period discarded. Your kernel does not support CPU cfs quota. Quota discarded.]
...
I1020 13:15:52.443195    3843 kubelet.go:1608] SyncLoop (DELETE, "api"): "rc-light-x68aj_e2e-tests-horizontal-pod-autoscaling-gtjuk(0465c18a-96c7-11e6-92eb-42010af00012)"

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-release-1.4/2708/

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Expected error:
    <*errors.errorString | 0xc820afd310>: {
        s: "error while stopping RC: rc-light: an error on the server (\"Internal Server Error: \\\"/api/v1/watch/namespaces/e2e-tests-horizontal-pod-autoscaling-j967i/replicationcontrollers?fieldSelector=metadata.name%3Drc-light&resourceVersion=8833\\\"\") has prevented the request from succeeding (get replicationControllers)",
    }
    error while stopping RC: rc-light: an error on the server ("Internal Server Error: \"/api/v1/watch/namespaces/e2e-tests-horizontal-pod-autoscaling-j967i/replicationcontrollers?fieldSelector=metadata.name%3Drc-light&resourceVersion=8833\"") has prevented the request from succeeding (get replicationControllers)
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:305

@k8s-github-robot k8s-github-robot mentioned this issue Nov 14, 2016
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Nov 22, 2016
Closed
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/314/

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Expected error:
    <*errors.errorString | 0xc820d92b10>: {
        s: "Only 1 pods started out of 2",
    }
    Only 1 pods started out of 2
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:345

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gci-gce/421/

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Nov 24 20:16:54.792: timeout waiting 15m0s for pods size to be 1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:286

This was referenced Nov 25, 2016
Closed
Closed
@jszczepkowski
Copy link
Contributor

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/ci-kubernetes-e2e-gke-staging-parallel/314/

Situation is similar to #33341 (comment), kubelet didn't start the pod, build logs:

I1123 09:56:13.568] Nov 23 09:50:41.602: INFO: Created replication controller with name: rc-light, namespace: e2e-tests-horizontal-pod-autoscaling-wijv5, replica count: 2
I1123 09:56:13.568] Nov 23 09:50:51.602: INFO: rc-light Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.568] Nov 23 09:51:01.603: INFO: rc-light Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.568] Nov 23 09:51:11.603: INFO: rc-light Pods: 2 out of 2 created, 0 running, 2 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.568] Nov 23 09:51:21.603: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:51:31.604: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:51:41.604: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:51:51.604: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:52:01.605: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:52:11.605: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:52:21.606: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:52:31.606: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.569] Nov 23 09:52:41.606: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.570] Nov 23 09:52:51.607: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.570] Nov 23 09:53:01.607: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.570] Nov 23 09:53:11.607: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.570] Nov 23 09:53:21.608: INFO: rc-light Pods: 2 out of 2 created, 1 running, 1 pending, 0 waiting, 0 inactive, 0 terminating, 0 unknown, 0 runningButNotReady 
I1123 09:56:13.570] Nov 23 09:53:21.608: INFO: Pod rc-light-3r4hq assigned to host gke-bootstrap-e2e-default-pool-823ca16b-roz6 (IP: 10.240.0.3) in Pending
I1123 09:56:13.570] Nov 23 09:53:21.608: INFO: 
I1123 09:56:13.570] Logging node info for node gke-bootstrap-e2e-default-pool-823ca16b-roz6
I1123 09:56:13.571] Nov 23 09:53:21.668: INFO: Node Info: &TypeMeta{Kind:,APIVersion:,}
I1123 09:56:13.571] Nov 23 09:53:21.668: INFO: 
I1123 09:56:13.571] Logging kubelet events for node gke-bootstrap-e2e-default-pool-823ca16b-roz6
I1123 09:56:13.571] Nov 23 09:53:21.707: INFO: 
I1123 09:56:13.571] Logging pods the kubelet thinks is on node gke-bootstrap-e2e-default-pool-823ca16b-roz6
I1123 09:56:13.571] W1123 09:53:21.719255    2312 metrics_grabber.go:73] Master node is not registered. Grabbing metrics from Scheduler and ControllerManager is disabled.
I1123 09:56:13.571] Nov 23 09:53:21.903: INFO: 
I1123 09:56:13.571] Latency metrics for node gke-bootstrap-e2e-default-pool-823ca16b-roz6
I1123 09:56:13.571] Nov 23 09:53:21.947: INFO: Pod rc-light-3r4hq	gke-bootstrap-e2e-default-pool-823ca16b-roz6	Pending	<nil>
I1123 09:56:13.572] Nov 23 09:53:21.947: INFO: Pod rc-light-hjuuy	gke-bootstrap-e2e-default-pool-823ca16b-i0m2	Running	<nil>
I1123 09:56:13.572] Nov 23 09:53:21.947: INFO: Unexpected error occurred: Only 1 pods started out of 2

kubelet-logs:

053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520413, nsec:970131297, loc:(*time.Location)(0x53053c0)}}, Count:2, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:53:36.176730   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520416, nsec:171049261, loc:(*time.Location)(0x53053c0)}}, Count:3, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:53:38.573911   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520418, nsec:570086407, loc:(*time.Location)(0x53053c0)}}, Count:4, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:53:42.974143   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520422, nsec:970081139, loc:(*time.Location)(0x53053c0)}}, Count:5, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:53:51.374531   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520431, nsec:370564846, loc:(*time.Location)(0x53053c0)}}, Count:6, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:54:07.775020   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520447, nsec:770542898, loc:(*time.Location)(0x53053c0)}}, Count:7, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:54:39.821256   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489beaeab05fd3c", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"MountVolume.SetUp failed for volume \"kubernetes.io/secret/59d8cce8-b1a5-11e6-ace1-42010af0000a-default-token-21n6x\" (spec.Name: \"default-token-21n6x\") pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" (UID: \"59d8cce8-b1a5-11e6-ace1-42010af0000a\") with: secrets \"default-token-21n6x\" not found", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520409, nsec:970801980, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520479, nsec:815510831, loc:(*time.Location)(0x53053c0)}}, Count:8, Type:"Warning"}': 'events "rc-light-3r4hq.1489beaeab05fd3c" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:55:26.723774   20348 kubelet.go:1813] Unable to mount volumes for pod "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "rc-light-3r4hq"/"e2e-tests-horizontal-pod-autoscaling-wijv5". list of unattached/unmounted volumes=[default-token-21n6x]; skipping pod
E1123 17:55:26.723847   20348 pod_workers.go:184] Error syncing pod 59d8cce8-b1a5-11e6-ace1-42010af0000a, skipping: timeout expired waiting for volumes to attach/mount for pod "rc-light-3r4hq"/"e2e-tests-horizontal-pod-autoscaling-wijv5". list of unattached/unmounted volumes=[default-token-21n6x]
E1123 17:55:26.725838   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489bec9da0c95da", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedSync", Message:"Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod \"rc-light-3r4hq\"/\"e2e-tests-horizontal-pod-autoscaling-wijv5\". list of unattached/unmounted volumes=[default-token-21n6x]", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520526, nsec:723880410, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520526, nsec:723880410, loc:(*time.Location)(0x53053c0)}}, Count:1, Type:"Warning"}': 'events "rc-light-3r4hq.1489bec9da0c95da" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
E1123 17:55:26.727481   20348 event.go:199] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"rc-light-3r4hq.1489bec9da0ab1e4", GenerateName:"", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:0, nsec:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]api.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:api.ObjectReference{Kind:"Pod", Namespace:"e2e-tests-horizontal-pod-autoscaling-wijv5", Name:"rc-light-3r4hq", UID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", APIVersion:"v1", ResourceVersion:"2920", FieldPath:""}, Reason:"FailedMount", Message:"Unable to mount volumes for pod \"rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)\": timeout expired waiting for volumes to attach/mount for pod \"rc-light-3r4hq\"/\"e2e-tests-horizontal-pod-autoscaling-wijv5\". list of unattached/unmounted volumes=[default-token-21n6x]", Source:api.EventSource{Component:"kubelet", Host:"gke-bootstrap-e2e-default-pool-823ca16b-roz6"}, FirstTimestamp:unversioned.Time{Time:time.Time{sec:63615520526, nsec:723756516, loc:(*time.Location)(0x53053c0)}}, LastTimestamp:unversioned.Time{Time:time.Time{sec:63615520526, nsec:723756516, loc:(*time.Location)(0x53053c0)}}, Count:1, Type:"Warning"}': 'events "rc-light-3r4hq.1489bec9da0ab1e4" is forbidden: unable to create new content in namespace e2e-tests-horizontal-pod-autoscaling-wijv5 because it is being terminated.' (will not retry!)
I1123 17:55:28.023901   20348 docker_manager.go:1492] Killing container "ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" with 30 second grace period
I1123 17:55:28.253364   20348 docker_manager.go:1531] Container "ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" exited after 229.427563ms
W1123 17:55:28.253428   20348 docker_manager.go:1537] No ref for pod '"ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq"'
I1123 17:55:28.859757   20348 docker_manager.go:1492] Killing container "06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" with 30 second grace period
I1123 17:55:28.981711   20348 docker_manager.go:1531] Container "06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" exited after 121.863572ms
W1123 17:55:28.981780   20348 docker_manager.go:1537] No ref for pod '"06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq"'
I1123 17:55:29.047521   20348 kubelet.go:2328] SyncLoop (PLEG): "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)", event: &pleg.PodLifecycleEvent{ID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", Type:"ContainerDied", Data:"ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77"}
I1123 17:55:29.047615   20348 kubelet.go:2328] SyncLoop (PLEG): "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)", event: &pleg.PodLifecycleEvent{ID:"59d8cce8-b1a5-11e6-ace1-42010af0000a", Type:"ContainerDied", Data:"06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f"}
I1123 17:55:29.130403   20348 kubelet.go:2309] SyncLoop (DELETE, "api"): "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)"
I1123 17:55:29.149351   20348 kubelet.go:2303] SyncLoop (REMOVE, "api"): "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5(59d8cce8-b1a5-11e6-ace1-42010af0000a)"
I1123 17:55:29.149471   20348 kubelet.go:2089] Killing unwanted pod "rc-light-3r4hq"
I1123 17:55:29.149540   20348 docker_manager.go:1492] Killing container "ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" with 0 second grace period
I1123 17:55:29.150631   20348 docker_manager.go:1531] Container "ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" exited after 1.065124ms
W1123 17:55:29.150663   20348 docker_manager.go:1537] No ref for pod '"ec9f3f4e0e4192713df1365832d0cf36d3c84e510bd45669982d6bf73e4abf77 rc-light e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq"'
I1123 17:55:29.181050   20348 docker_manager.go:1492] Killing container "06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" with 0 second grace period
I1123 17:55:29.181719   20348 docker_manager.go:1531] Container "06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq" exited after 631.545µs
W1123 17:55:29.181755   20348 docker_manager.go:1537] No ref for pod '"06e6a9da044e414266301eff5653760be05aeb5f39027572e277496a9ffc4a1f e2e-tests-horizontal-pod-autoscaling-wijv5/rc-light-3r4hq"'
E1123 17:55:29.181801   20348 kubelet.go:2092] Failed killing the pod "rc-light-3r4hq": failed to "TeardownNetwork" for "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5" with TeardownNetworkError: "Failed to teardown network for pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" using network plugins \"kubenet\": Error removing container from network: failed to Statfs \"/proc/0/ns/net\": no such file or directory"
I1123 17:55:29.717939   20348 kubelet.go:2089] Killing unwanted pod "rc-light-3r4hq"
E1123 17:55:29.805295   20348 kubelet.go:2092] Failed killing the pod "rc-light-3r4hq": failed to "TeardownNetwork" for "rc-light-3r4hq_e2e-tests-horizontal-pod-autoscaling-wijv5" with TeardownNetworkError: "Failed to teardown network for pod \"59d8cce8-b1a5-11e6-ace1-42010af0000a\" using network plugins \"kubenet\": Error removing container from network: failed to Statfs \"/proc/0/ns/net\": no such file or directory"
~

@k8s-github-robot
Copy link
Author

k8s-github-robot commented Nov 25, 2016 •
edited by rmmh

Builds:
ci-kubernetes-e2e-gci-gce-proto 791
ci-kubernetes-e2e-gci-gke 392

Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 2 pods to 1 pod {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/horizontal_pod_autoscaling.go:100
Nov 25 10:35:02.199: timeout waiting 15m0s for pods size to be 1
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/autoscaling_utils.go:286

This was referenced Nov 28, 2016
Closed
Closed
@jszczepkowski jszczepkowski mentioned this issue Nov 30, 2016
Closed
This was referenced Feb 27, 2017
Closed
Closed
Closed
@calebamiles calebamiles added the sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. label Feb 28, 2017
@calebamiles calebamiles modified the milestone: v1.6 Mar 3, 2017
@jszczepkowski
Copy link
Contributor

The bug was fixed on 1.5 branch. Failed runs refers to release 1.4 (or earlier).

Closing as obsolete.

This was referenced Mar 10, 2017
Closed
Closed
Closed
This was referenced Mar 18, 2017
Closed
Closed
Closed
Closed
Closed
Closed
Closed
This was referenced Mar 27, 2017
Closed
Closed
Closed
Closed
Closed
Closed
Closed
@k8s-github-robot k8s-github-robot mentioned this issue Apr 22, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vishh vishh

@dchen1107 dchen1107

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Projects
None yet
Milestone
v1.6
Development

No branches or pull requests
5 participants
@calebamiles @vishh @dchen1107 @jszczepkowski @k8s-github-robot