-
Notifications
You must be signed in to change notification settings - Fork 39k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kubernetes-e2e-gke: broken test run #27839
Comments
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9675/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9676/ Run so broken it didn't make JUnit output! |
Are we having trouble reading the git version?
|
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9682/ Multiple broken tests: Failed: [k8s.io] Deployment deployment should support rollover {Kubernetes e2e suite}
Issues about this test specifically: #26509 #26834 Failed: [k8s.io] Kubectl client [k8s.io] Update Demo should create and stop a replication controller [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Probing container with readiness probe that fails should never be ready and never restart [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] NodeProblemDetector [k8s.io] KernelMonitor should generate node condition and events for corresponding errors {Kubernetes e2e suite}
Failed: [k8s.io] SSH should SSH to all nodes and run commands {Kubernetes e2e suite}
Issues about this test specifically: #26129 Failed: [k8s.io] Pods should be updated [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] ConfigMap should be consumable from pods in volume as non-root [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27245 |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9683/ Multiple broken tests: Failed: [k8s.io] Downward API volume should update annotations on modification [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Service endpoints latency should not be very high [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support inline execution and attach {Kubernetes e2e suite}
Issues about this test specifically: #26324 #27715 Failed: [k8s.io] V1Job should scale a job up {Kubernetes e2e suite}
Failed: [k8s.io] Deployment RollingUpdateDeployment should scale up and down in the right order {Kubernetes e2e suite}
Issues about this test specifically: #27232 Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27532 Failed: [k8s.io] V1Job should run a job to completion when tasks succeed {Kubernetes e2e suite}
Failed: [k8s.io] Deployment deployment should create new pods {Kubernetes e2e suite}
Failed: [k8s.io] EmptyDir volumes volume on tmpfs should have the correct mode [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl label should update the label on a resource [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Job should scale a job down {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a persistent volume claim. {Kubernetes e2e suite}
Failed: [k8s.io] Variable Expansion should allow composing env vars into new env vars [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl apply should apply a new configuration to an existing RC {Kubernetes e2e suite}
Issues about this test specifically: #27524 Failed: [k8s.io] ConfigMap should be consumable from pods in volume with mappings as non-root [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should verify ResourceQuota with terminating scopes. {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a replication controller. {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run pod should create a pod from an image when restart is Never [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27507 Failed: [k8s.io] ReplicaSet should serve a basic image on each replica with a private image {Kubernetes e2e suite}
Failed: [k8s.io] Deployment RollingUpdateDeployment should delete old pods and create new ones {Kubernetes e2e suite}
Failed: [k8s.io] Probing container with readiness probe that fails should never be ready and never restart [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] V1Job should scale a job down {Kubernetes e2e suite}
Failed: [k8s.io] Proxy version v1 should proxy logs on node with explicit kubelet port [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Update Demo should scale a replication controller [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run job should create a job from an image when restart is OnFailure [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Networking should provide unchanging, static URL paths for kubernetes api services [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26838 Failed: [k8s.io] KubeletManagedEtcHosts should test kubelet managed /etc/hosts file {Kubernetes e2e suite}
Issues about this test specifically: #27023 Failed: [k8s.io] [HPA] Horizontal pod autoscaling (scale resource: CPU) [k8s.io] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Issues about this test specifically: #27443 #27835 Failed: [k8s.io] Deployment deployment should support rollback when there's replica set with no revision {Kubernetes e2e suite}
Failed: [k8s.io] ConfigMap updates should be reflected in volume [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Generated release_1_2 clientset should create pods, delete pods, watch pods {Kubernetes e2e suite}
Failed: [k8s.io] ConfigMap should be consumable from pods in volume as non-root [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27245 Failed: [k8s.io] Pods should support retrieving logs from the container over websockets {Kubernetes e2e suite}
Failed: [k8s.io] Pods should be restarted with a docker exec "cat /tmp/health" liveness probe [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Mesos starts static pods on every node in the mesos cluster {Kubernetes e2e suite}
Failed: [k8s.io] Services should create endpoints for unready pods {Kubernetes e2e suite}
Issues about this test specifically: #26172 Failed: [k8s.io] Services should provide secure master service [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Variable Expansion should allow substituting values in a container's args [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and ensure its status is promptly calculated. {Kubernetes e2e suite}
|
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9685/ Multiple broken tests: Failed: [k8s.io] PreStop should call prestop when killing a pod [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Services should be able to create a functioning NodePort service {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26209 Failed: [k8s.io] EmptyDir volumes should support (non-root,0666,default) [Conformance] {Kubernetes e2e suite}
|
One another from #27704: Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}
|
It's like halfway through the test run the apiserver starts returning 403 unauthorized for all apiserver resources. Unfortunately there are no master logs to help debug. cc @kubernetes/goog-gke any ideas? |
/cc @cjcullen We've seen forbidden responses causing errors on the following recent test runs: 9685 9683 9682 9676 9675 9674 9671 |
Thought this was fixed, but I guess fix didn't take. See #26742 (comment) |
When did that change go in? The last series of errors we saw was between ~1am and 2:30am this morning. |
@mikedanese sure there are master logs, you just have to do about 30 seconds of digging through bigquery for them. The short hash for the first post's cluster was c5aed2ce |
The webhook authorizer/authenticator isn't checking the response status code, so it ends up caching our backend flakes :(. |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9708/ Multiple broken tests: Failed: [k8s.io] Variable Expansion should allow substituting values in a container's command [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Job should run a job to completion when tasks succeed {Kubernetes e2e suite}
Failed: [k8s.io] ReplicationController should serve a basic image on each replica with a private image {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a replication controller. {Kubernetes e2e suite}
Failed: [k8s.io] V1Job should run a job to completion when tasks sometimes fail and are locally restarted {Kubernetes e2e suite}
Failed: [k8s.io] Services should serve a basic endpoint from pods [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26678 Failed: [k8s.io] Services should provide secure master service [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
Failed: [k8s.io] EmptyDir volumes should support (root,0666,tmpfs) [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Downward API should provide container's limits.cpu/memory and requests.cpu/memory as env vars {Kubernetes e2e suite}
Failed: [k8s.io] Networking should function for intra-pod communication [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26960 #27235 Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27532 Failed: [k8s.io] Monitoring should verify monitoring pods and all cluster nodes are available on influxdb using heapster. {Kubernetes e2e suite}
Failed: [k8s.io] Pods should invoke init containers on a RestartAlways pod {Kubernetes e2e suite}
Failed: [k8s.io] Downward API volume should update labels on modification [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl apply should apply a new configuration to an existing RC {Kubernetes e2e suite}
Issues about this test specifically: #27524 |
@a-robinson that's a good tip. @cjcullen mind if I hand this off to you since you have context? let me know if there's anything I can do. |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10652/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10653/ Run so broken it didn't make JUnit output! |
This is broken again. From api server log
Api server is crashing. @lavalamp Did we recently remove the top-level panic recover? cc: @jlowdermilk |
Seems that we have. The console of the e2e is:
Which seems to indicate a crash followed by periods of unavailability until the apiserver restarts. |
I'm guessing #28520 is the culprit. |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10654/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10655/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10656/ Run so broken it didn't make JUnit output! |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10664/ Multiple broken tests: Failed: [k8s.io] Kubectl client [k8s.io] Update Demo should create and stop a replication controller [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Guestbook application should create and stop a working application [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26175 #26846 #27334 #28293 Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support inline execution and attach {Kubernetes e2e suite}
Issues about this test specifically: #26324 #27715 Failed: [k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26209 |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10667/ Multiple broken tests: Failed: [k8s.io] PrivilegedPod should test privileged pod {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support inline execution and attach {Kubernetes e2e suite}
Issues about this test specifically: #26324 #27715 Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support port-forward {Kubernetes e2e suite}
Issues about this test specifically: #28371 Failed: [k8s.io] Kubectl client [k8s.io] Update Demo should scale a replication controller [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #28437 |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10670/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10677/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10684/ Run so broken it didn't make JUnit output! |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10688/ Run so broken it didn't make JUnit output! |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10705/ Multiple broken tests: Failed: [k8s.io] Kubectl client [k8s.io] Kubectl api-versions should check if v1 is in available api versions [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Port forwarding [k8s.io] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27680 Failed: [k8s.io] Kubectl client [k8s.io] Kubectl patch should add annotations for pods in rc [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26126 Failed: [k8s.io] Kubectl client [k8s.io] Kubectl label should update the label on a resource [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #28493 |
Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10711/ Run so broken it didn't make JUnit output! |
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/9674/
Multiple broken tests:
Failed: [k8s.io] V1Job should scale a job down {Kubernetes e2e suite}
Failed: [k8s.io] Pods should get a host IP [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Generated release_1_3 clientset should create pods, delete pods, watch pods {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run deployment should create a deployment from an image [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #27532
Failed: [k8s.io] Pods should be restarted with a /healthz http liveness probe [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Simple pod should support exec through an HTTP proxy {Kubernetes e2e suite}
Issues about this test specifically: #27156
Failed: [k8s.io] ResourceQuota should verify ResourceQuota with terminating scopes. {Kubernetes e2e suite}
Failed: [k8s.io] V1Job should run a job to completion when tasks sometimes fail and are locally restarted {Kubernetes e2e suite}
Failed: [k8s.io] Probing container with readiness probe should not be ready before initial delay and never restart [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Variable Expansion should allow substituting values in a container's command [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Job should scale a job up {Kubernetes e2e suite}
Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}
Issues about this test specifically: #27704
Failed: [k8s.io] ReplicaSet should serve a basic image on each replica with a private image {Kubernetes e2e suite}
Failed: [k8s.io] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
Failed: [k8s.io] DNS should provide DNS for services [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26168 #27450
Failed: [k8s.io] MetricsGrabber should grab all metrics from a ControllerManager. {Kubernetes e2e suite}
Failed: [k8s.io] Downward API volume should provide container's cpu request {Kubernetes e2e suite}
Failed: [k8s.io] Downward API should provide pod IP as an env var {Kubernetes e2e suite}
Failed: [k8s.io] Downward API should provide default limits.cpu/memory from node capacity {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a pod. {Kubernetes e2e suite}
Failed: [k8s.io] Services should be able to create a functioning NodePort service {Kubernetes e2e suite}
Failed: ThirdParty resources Simple Third Party creating/deleting thirdparty objects works [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] EmptyDir volumes should support (root,0777,default) [Conformance] {Kubernetes e2e suite}
Issues about this test specifically: #26780
Failed: [k8s.io] Deployment deployment should create new pods {Kubernetes e2e suite}
Failed: [k8s.io] EmptyDir volumes should support (non-root,0666,tmpfs) [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] PreStop should call prestop when killing a pod [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl version should check is all data is printed [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] DNS should provide DNS for pods for Hostname and Subdomain Annotation {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Update Demo should create and stop a replication controller [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Job should run a job to completion when tasks succeed {Kubernetes e2e suite}
Failed: [k8s.io] Downward API should provide pod name and namespace as env vars [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Downward API volume should update annotations on modification [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] SSH should SSH to all nodes and run commands {Kubernetes e2e suite}
Issues about this test specifically: #26129
Failed: [k8s.io] Services should check NodePort out-of-range {Kubernetes e2e suite}
Failed: [k8s.io] ResourceQuota should create a ResourceQuota and capture the life of a persistent volume claim. {Kubernetes e2e suite}
Failed: [k8s.io] Job should delete a job {Kubernetes e2e suite}
Failed: [k8s.io] Job should run a job to completion when tasks sometimes fail and are not locally restarted {Kubernetes e2e suite}
Failed: [k8s.io] Services should be able to up and down services {Kubernetes e2e suite}
Issues about this test specifically: #26128 #26685
Failed: [k8s.io] Pods should contain environment variables for services [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl run rc should create an rc from an image [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Probing container with readiness probe that fails should never be ready and never restart [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] Kubectl client [k8s.io] Kubectl api-versions should check if v1 is in available api versions [Conformance] {Kubernetes e2e suite}
Failed: [k8s.io] ReplicationController should serve a basic image on each replica with a private image {Kubernetes e2e suite}
Previous issues for this suite: #26742
The text was updated successfully, but these errors were encountered: