[k8s.io] V1Job should fail a job {Kubernetes e2e suite}

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/5563/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:213
Expected error:
    <*errors.StatusError | 0xc8209f3d80>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on jobs.batch \"foo\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {Name: "foo", Group: "batch", Kind: "jobs", Causes: nil, RetryAfterSeconds: 0},
            Code: 409,
        },
    }
    Operation cannot be fulfilled on jobs.batch "foo": the object has been modified; please apply your changes to the latest version and try again
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:209

Previous issues for this test: #27704 #30127 #30602 #31070

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gce-etcd3/3806/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:213
Expected error:
    <*errors.StatusError | 0xc8209a2b00>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on jobs.batch \"foo\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {Name: "foo", Group: "batch", Kind: "jobs", Causes: nil, RetryAfterSeconds: 0},
            Code: 409,
        },
    }
    Operation cannot be fulfilled on jobs.batch "foo": the object has been modified; please apply your changes to the latest version and try again
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:209

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/6067/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:213
Expected error:
    <*errors.StatusError | 0xc420368580>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on jobs.batch \"foo\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {Name: "foo", Group: "batch", Kind: "jobs", Causes: nil, RetryAfterSeconds: 0},
            Code: 409,
        },
    }
    Operation cannot be fulfilled on jobs.batch "foo": the object has been modified; please apply your changes to the latest version and try again
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:209

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/18861/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:213
Expected error:
    <*errors.StatusError | 0xc420219080>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on jobs.batch \"foo\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {Name: "foo", Group: "batch", Kind: "jobs", Causes: nil, RetryAfterSeconds: 0},
            Code: 409,
        },
    }
    Operation cannot be fulfilled on jobs.batch "foo": the object has been modified; please apply your changes to the latest version and try again
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:209

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/6768/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
Oct 31 19:32:54.023: Couldn't delete ns: "e2e-tests-v1job-v59bf": namespace e2e-tests-v1job-v59bf was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed (&errors.errorString{s:"namespace e2e-tests-v1job-v59bf was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gci-gce-etcd3/1879/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:213
Expected error:
    <*errors.StatusError | 0xc42134d700>: {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {SelfLink: "", ResourceVersion: ""},
            Status: "Failure",
            Message: "Operation cannot be fulfilled on jobs.batch \"foo\": the object has been modified; please apply your changes to the latest version and try again",
            Reason: "Conflict",
            Details: {Name: "foo", Group: "batch", Kind: "jobs", Causes: nil, RetryAfterSeconds: 0},
            Code: 409,
        },
    }
    Operation cannot be fulfilled on jobs.batch "foo": the object has been modified; please apply your changes to the latest version and try again
not to have occurred
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/batch_v1_jobs.go:209

[k8s-github-robot]

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-aws/7302/

Failed: [k8s.io] V1Job should fail a job {Kubernetes e2e suite}

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:142
Nov 11 09:47:58.136: Couldn't delete ns: "e2e-tests-v1job-6vufx": namespace e2e-tests-v1job-6vufx was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed (&errors.errorString{s:"namespace e2e-tests-v1job-6vufx was not deleted with limit: timed out waiting for the condition, namespace is empty but is not yet removed"})
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/framework/framework.go:353

[krousey]

@soltysh These are now failing in upgrade tests. Could you take a look.

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke-gci-1.3-gci-1.5-upgrade-cluster/405

[soltysh]

Yup, I'll try to look over the weekend.

[soltysh]

@krousey looking through the logs it looks like this failure is happening in 1.4 (matching lines). I've fixed this problem in 1.5 in #37077. Can you confirm that? Not sure how the upgrade tests is run currently, could you point me to the flow? Maybe we should cherry-pick that PR to 1.4?

[saad-ali]

To prevent the 1.5 release from slipping, we need all blocker issues investigated and fixed or triaged as non-blocking ASAP. So, @soltysh, I'd like to get an update on this issue for the 1.5 release. Is it still considered a blocker? If so, what is needed to fix it?

[soltysh]

I'm still waiting for @krousey input on that one. I've pointed in my previous comment what might be the cause and didn't get answer yet. @krousey can you confirm my observations from my previous comment? Is this still a release blocker?

[krousey]

@soltysh I could not find an incident of this failing in 1.4. You're welcome to double check https://k8s-testgrid.appspot.com/release-1.4-blocking#gce-release-1.4&sort-by-failures=. As far as the upgrade flow, it's like this:

• *-upgrade-master:

  1. Turn up cluster at old version.
  2. Upgrade the master to new version.
  3. Run e2e tests at old version.

• *-upgrade-cluster:

  1. Turn up cluster at old version.
  2. Upgrade master and nodes to new version.
  3. Run e2e tests at old version.

• *-upgrade-cluster-new:

  1. Turn up cluster at old version.
  2. Upgrade master and nodes to new version.
  3. Run e2e tests at new version.

This seems to fit with your fix to the test, as I don't see this failure in *-cluster-new suites.

[calebamiles]

Thanks for the update @krousey taken with the comments from @soltysh can we mark this as a non release blocking issue for 1.5? Will we need a cherrypick of the fix to 1.4?

cc: @saad-ali, @dims, @jessfraz

[krousey]

This will continue to fail in the upgrade tests (except for cluster-new ones) and it is understood to be a test issue and not a product issue. I vote it is a non-release blocker that should probably be cherry-picked to 1.3 and 1.4.

[saad-ali]

This will continue to fail in the upgrade tests (except for cluster-new ones) and it is understood to be a test issue and not a product issue. I vote it is a non-release blocker that should probably be cherry-picked to 1.3 and 1.4.

Marking non-release-blocking

[dims]

@soltysh Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)

[soltysh]

Test was removed in #40845.