Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add remaining e2e tests for Job BackoffLimitPerIndex based on KEP #121633

Merged
merged 1 commit into from Nov 2, 2023
Merged

Add remaining e2e tests for Job BackoffLimitPerIndex based on KEP #121633

merged 1 commit into from Nov 2, 2023

Conversation

mimowo
Copy link
Contributor

@mimowo mimowo commented Oct 31, 2023

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Which issue(s) this PR fixes:

Part of: kubernetes/enhancements#3850

Special notes for your reviewer:

The test scenarios were outlined in the KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3850-backoff-limits-per-index-for-indexed-jobs, but I forgot about them when preparing #121368. Filling the gap now.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3850-backoff-limits-per-index-for-indexed-jobs

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 31, 2023
@k8s-ci-robot k8s-ci-robot added area/test sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 31, 2023
@mimowo mimowo changed the title Add remaining e2e tests for backoffLimitPerIndex based on KEP Add remaining e2e tests for Job BackoffLimitPerIndex based on KEP Oct 31, 2023
@mimowo
Copy link
Contributor Author

mimowo commented Oct 31, 2023

/sig apps
/assign @alculquicondor
FYI @pohly , @kannon92

alculquicondor
alculquicondor reviewed Oct 31, 2023
View reviewed changes
test/e2e/apps/job.go
// we use parallelism=1 to make sure in the asserts only one pod was created
parallelism = int32(1)
ginkgo.By("Creating an indexed job with backoffLimit per index and maxFailedIndexes")
job := e2ejob.NewTestJob("fail", "with-max-failed-indexes", v1.RestartPolicyNever, parallelism, completions, nil, backoffLimit)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

side note: having these global variables for parallelism, completions and backoffLimit make the tests very hard to read. I need to jump from one place to another to check what the values are. We should get rid of them in a follow up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

Copy link
Contributor Author

@mimowo mimowo Nov 2, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I just realized this parallelism = int32(1) was a bug that could override the parallelism for the next test to run. This caused this failure which I also reproduced locally: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/121633/pull-kubernetes-e2e-gce/1719394173829779456. It could cause flakes depending on the order of execution of the tests.

Fixed now by parallelism := int32(1). However, its better to avoid the global vars. I will open an issue for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened the cleanup issue for now, maybe some folks will want to grab: #121689

test/e2e/apps/job.go Outdated
ginkgo.By("Verifying the Job status fields to ensure the upper indexes didn't execute")
job, err = e2ejob.GetJob(ctx, f.ClientSet, f.Namespace.Name, job.Name)
framework.ExpectNoError(err, "failed to retrieve latest job object")
gomega.Expect(job.Status.FailedIndexes).Should(gomega.HaveValue(gomega.Equal("0,1")))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a guarantee that both pods will run? Could this flake?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a guarantee on that, because the job does not specify maxFailedIndexes, so all indexes are expected to be executed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran it a couple of times locally and it passed consistently.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, right, it's a FailIndex, not FailJob.
It would be more interesting to use a job where (let's say, odd numbers fail). I think there's a helper for that already?

Copy link
Contributor Author

@mimowo mimowo Oct 31, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, we have a helper, I was thinking about using this. I will adjust if you think it makes more sense.

@mimowo mimowo force-pushed the backoff-limit-per-index-remaining-e2e-test branch from e5fd24e to cb18d82 Compare October 31, 2023 16:41
@mimowo
Copy link
Contributor Author

mimowo commented Oct 31, 2023

/test pull-kubernetes-unit
unrelated failure

alculquicondor
alculquicondor reviewed Oct 31, 2023
View reviewed changes
Copy link
Member

@alculquicondor alculquicondor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve
/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 31, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: c74bf0c2f313f25cbd9f9c1008e540d3f088a6c8

@mimowo
Copy link
Contributor Author

mimowo commented Oct 31, 2023

/retest

kannon92
kannon92 reviewed Oct 31, 2023
View reviewed changes
Copy link
Contributor

@kannon92 kannon92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@alculquicondor
Copy link
Member

/assign @soltysh

@mimowo mimowo force-pushed the backoff-limit-per-index-remaining-e2e-test branch from cb18d82 to ae73cf9 Compare November 2, 2023 07:23
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 2, 2023
@mimowo
Copy link
Contributor Author

mimowo commented Nov 2, 2023

/assign @soltysh
The LGTM was removed due to the fix described here: #121633 (comment)

@mimowo mimowo mentioned this pull request Nov 2, 2023
Closed
@mimowo
Copy link
Contributor Author

mimowo commented Nov 2, 2023

/test pull-kubernetes-unit

@mimowo
Copy link
Contributor Author

mimowo commented Nov 2, 2023

@pacoxu
Could you add the 1.29 milestone here?

@HuSharp HuSharp mentioned this pull request Nov 2, 2023
Merged
soltysh
soltysh approved these changes Nov 2, 2023
View reviewed changes
Copy link
Contributor

@soltysh soltysh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@soltysh
Copy link
Contributor

soltysh commented Nov 2, 2023

/triage accepted
/priority backlog

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Nov 2, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, mimowo, soltysh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2023
@soltysh
Copy link
Contributor

soltysh commented Nov 2, 2023

/milestone v1.29
/lgtm

@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Nov 2, 2023
@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 2, 2023
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: c7fb7c87a8bdc4246388d325a23467eea10db4c3

@jensentanlo jensentanlo mentioned this pull request Nov 2, 2023
Open
8 tasks
@k8s-ci-robot k8s-ci-robot merged commit 6eee80f into kubernetes:master Nov 2, 2023
15 checks passed
@mimowo mimowo deleted the backoff-limit-per-index-remaining-e2e-test branch November 29, 2023 15:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kannon92 kannon92 kannon92 left review comments

@soltysh soltysh soltysh approved these changes

@atiratree atiratree Awaiting requested review from atiratree

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@alculquicondor alculquicondor Awaiting requested review from alculquicondor

Assignees

@soltysh soltysh

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/backlog Higher priority than priority/awaiting-more-evidence. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
SIG Apps
Archived in project
[sig-release] Bug Triage
Archived in project
Milestone
v1.29
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@mimowo @k8s-ci-robot @alculquicondor @soltysh @kannon92