Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kubernetes-e2e-gci-gke-subnet: broken test run #34069

Closed
k8s-github-robot opened this issue Oct 5, 2016 · 9 comments
Closed

kubernetes-e2e-gci-gke-subnet: broken test run #34069

k8s-github-robot opened this issue Oct 5, 2016 · 9 comments
Assignees
@jsafrane
Labels
area/test-infra kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-subnet/58/

Multiple broken tests:

Failed: DiffResources {e2e.go}

Error: 1 leaked resources
+gke-auto-subnet-4ecc4d-pvc-440ff4c7-8a86-11e6-9d4d-42010af00058  us-central1-f  2        pd-standard  READY

Issues about this test specifically: #33373 #33416 #34060

Failed: [k8s.io] SchedulerPredicates [Serial] validates that required NodeAffinity setting is respected if matching {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233
Expected error:
    <*errors.errorString | 0xc8223bd960>: {
        s: "Namespace e2e-tests-volume-provisioning-aj2fi is active",
    }
    Namespace e2e-tests-volume-provisioning-aj2fi is active
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

Issues about this test specifically: #28071

Failed: [k8s.io] SchedulerPredicates [Serial] validates that taints-tolerations is respected if matching {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233
Expected error:
    <*errors.errorString | 0xc82193b310>: {
        s: "Namespace e2e-tests-volume-provisioning-aj2fi is active",
    }
    Namespace e2e-tests-volume-provisioning-aj2fi is active
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

Issues about this test specifically: #28853 #31585

Failed: [k8s.io] SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233
Expected error:
    <*errors.errorString | 0xc82183a0d0>: {
        s: "Namespace e2e-tests-volume-provisioning-aj2fi is active",
    }
    Namespace e2e-tests-volume-provisioning-aj2fi is active
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

Issues about this test specifically: #33883

Failed: [k8s.io] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233
Expected error:
    <*errors.errorString | 0xc8224336b0>: {
        s: "Namespace e2e-tests-volume-provisioning-aj2fi is active",
    }
    Namespace e2e-tests-volume-provisioning-aj2fi is active
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

Issues about this test specifically: #29816 #30018 #33974

Failed: [k8s.io] SchedulerPredicates [Serial] validates that InterPod Affinity and AntiAffinity is respected if matching {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:233
Expected error:
    <*errors.errorString | 0xc820a3f430>: {
        s: "Namespace e2e-tests-volume-provisioning-aj2fi is active",
    }
    Namespace e2e-tests-volume-provisioning-aj2fi is active
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/scheduler_predicates.go:211

Failed: [k8s.io] Kubelet [Serial] [Slow] [k8s.io] regular resource usage tracking resource tracking for 100 pods per node {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:278
Oct  4 13:43:11.120: CPU usage exceeding limits:
 node gke-auto-subnet-default-pool-8676cabe-ljpg:
 container "kubelet": expected 95th% usage < 0.220; got 0.259
node gke-auto-subnet-default-pool-8676cabe-kab1:
 container "kubelet": expected 50th% usage < 0.170; got 0.175, container "kubelet": expected 95th% usage < 0.220; got 0.233
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/kubelet_perf.go:187

Issues about this test specifically: #26982 #32214 #33994 #34035

Failed: [k8s.io] Dynamic provisioning [k8s.io] DynamicProvisioner should create and delete persistent volumes [Slow] {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:135
Expected error:
    <*errors.StatusError | 0xc8208b8180>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "the server has asked for the client to provide credentials (delete persistentVolumeClaims pvc-5is7k)",
            Reason: "Unauthorized",
            Details: {
                Name: "pvc-5is7k",
                Group: "",
                Kind: "persistentVolumeClaims",
                Causes: [
                    {
                        Type: "UnexpectedServerResponse",
                        Message: "Unauthorized",
                        Field: "",
                    },
                ],
                RetryAfterSeconds: 0,
            },
            Code: 401,
        },
    }
    the server has asked for the client to provide credentials (delete persistentVolumeClaims pvc-5is7k)
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/volume_provisioning.go:98

Issues about this test specifically: #32185 #32372

Previous issues for this suite: #33404
@k8s-github-robot k8s-github-robot added kind/flake Categorizes issue or PR as related to a flaky test. priority/backlog Higher priority than priority/awaiting-more-evidence. area/test-infra labels Oct 5, 2016
@ixdy ixdy assigned deads2k and unassigned ixdy Oct 5, 2016
@ixdy
Copy link
Member

ixdy commented Oct 5, 2016

Root issue seems to be the failing PVC test.

@deads2k deads2k assigned jsafrane and unassigned deads2k Oct 5, 2016
@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-subnet/-1/

Run so broken it didn't make JUnit output!

@jsafrane
Copy link
Member

jsafrane commented Oct 6, 2016

There are no logs from controller-manager and the test failed with

Oct  4 16:05:40.234: INFO: Unexpected error occurred: the server has asked for the client to provide credentials (delete persistentVolumeClaims pvc-5is7k)

I.e. the test was not authorized to delete a PVC. I know very little about authentication/authorization and it seems odd to me that the test could create namespace/pvc/pods and can't delete it. @deads2k, does it ring any bell?

@deads2k
Copy link
Contributor

deads2k commented Oct 7, 2016

I.e. the test was not authorized to delete a PVC. I know very little about authentication/authorization and it seems odd to me that the test could create namespace/pvc/pods and can't delete it. @deads2k, does it ring any bell?

Asking for credentials means 401 (authentication). Any chance you're using a service account token for a namespace that the namespace controller is cleaning up underneath you?

@jsafrane
Copy link
Member

jsafrane commented Oct 7, 2016

Any chance you're using a service account token for a namespace that the namespace controller is cleaning up underneath you?

No, the test uses standard e2e framework where a namespace is created at the beginning and it's destroyed after the test finishes. Almost the same test "DynamicProvisioner Alpha" was successful in this test run - it calls the same testDynamicProvisioning as the failed test, just with slightly different objects. Compare https://github.com/jsafrane/kubernetes/blob/master/test/e2e/volume_provisioning.go#L116 (failed) with https://github.com/jsafrane/kubernetes/blob/master/test/e2e/volume_provisioning.go#L138 (succeeded).

@deads2k
Copy link
Contributor

deads2k commented Oct 7, 2016

Well, you'll only end up with a 401 if you're submitting an invalid token and SA tokens are usually the culprit. Not sure how you'd get it otherwise.

@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-subnet/78/

Run so broken it didn't make JUnit output!

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Oct 11, 2016
@k8s-github-robot
Copy link
Author

Failed: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gke-subnet/84/

Run so broken it didn't make JUnit output!

@k8s-github-robot k8s-github-robot added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Oct 11, 2016
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @jsafrane

This flaky-test issue would love to have more attention.

@fejta fejta closed this as completed Oct 12, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsafrane jsafrane

Labels
area/test-infra kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@fejta @jsafrane @ixdy @deads2k @k8s-github-robot