Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement backoff re-queuing mechanism #1709

Merged
merged 13 commits into from Feb 12, 2024
Merged

Implement backoff re-queuing mechanism #1709

merged 13 commits into from Feb 12, 2024

Conversation

tenzen-y
Copy link
Member

@tenzen-y tenzen-y commented Feb 9, 2024 •
edited by k8s-ci-robot

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

I implemented a backoff re-queue mechanism when a workload meets an Eviction with PodsReadyTimout reason.

Which issue(s) this PR fixes:

Part-of #1282

Special notes for your reviewer:

KEP-1282: https://github.com/kubernetes-sigs/kueue/blob/3cd0da66f57dc90f9d5e631b218f7a374204d07d/keps/1282-pods-ready-requeue-strategy/README.md

Does this PR introduce a user-facing change?

WaitForPodsReady: Support a backoff re-queueing mechanism with configurable limit.

@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Feb 9, 2024
@netlify Netlify
Copy link

netlify bot commented Feb 9, 2024 •
edited

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name Link
🔨 Latest commit 3bd7ce6
🔍 Latest deploy log https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65ca79e7ff0cd90008be03a2

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 9, 2024
@tenzen-y tenzen-y marked this pull request as draft February 9, 2024 08:51
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 9, 2024
@tenzen-y tenzen-y changed the title Implement backoff requeue machanism WIP: Implement backoff requeue machanism Feb 9, 2024
@tenzen-y tenzen-y force-pushed the implement-backoff-requeue-machanism branch 9 times, most recently from 1f78bf3 to fede9c9 Compare February 9, 2024 22:14
@k8s-ci-robot k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Feb 9, 2024
@tenzen-y tenzen-y force-pushed the implement-backoff-requeue-machanism branch 6 times, most recently from 6054a7d to b2022b0 Compare February 10, 2024 02:25
@tenzen-y
Just checking if the workload has an evicted condition
37efbbf 
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@tenzen-y tenzen-y force-pushed the implement-backoff-requeue-machanism branch from 4fe812d to 37efbbf Compare February 12, 2024 19:06
@tenzen-y
Copy link
Member Author

Rebased

@tenzen-y
Record events after the workload is updated
df45ca4 
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@alculquicondor
Copy link
Contributor

Can you fix the verify?

@tenzen-y
Copy link
Member Author

Can you fix the verify?

Sure.

@alculquicondor
Copy link
Contributor

LGTM otherwise.

@tenzen-y
Rename canNotQueueWorkloadByBackoff to backoffWaitingTimeExpired
7d960ef 
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
alculquicondor
alculquicondor reviewed Feb 12, 2024
View reviewed changes
Copy link
Contributor

@alculquicondor alculquicondor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave for a follow up to remove the stub. I don't think it's necessary, if we rework the unit tests.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 3f48edfa9fdae121b674d87983ed3aeae76ff5ab

@tenzen-y
Always insert requeueState
6e65d23 
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2024
@tenzen-y
Copy link
Member Author

I'm still working on #1709 (comment) and #1709 (comment).

@tenzen-y
Remove not useful api comments
795d064 
Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@alculquicondor
Copy link
Contributor

/hold

@tenzen-y
Avoid flakinesses of integration tests caused by checking the backoff…
3bd7ce6 
… pending

Signed-off-by: Yuki Iwai <yuki.iwai.tz@gmail.com>
@tenzen-y
Copy link
Member Author

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 12, 2024
@tenzen-y
Copy link
Member Author

tenzen-y commented Feb 12, 2024 •
edited

Let's leave for a follow up to remove the stub. I don't think it's necessary, if we rework the unit tests.

/lgtm

Let me try to rework the unit tests to avoid stubs as a follow-up.

@alculquicondor
Copy link
Contributor

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 12, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 7113c7a278f1978076e5616e526705aa6ca25257

@k8s-ci-robot k8s-ci-robot merged commit 5a0a714 into kubernetes-sigs:main Feb 12, 2024
14 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v0.6 milestone Feb 12, 2024
@tenzen-y tenzen-y deleted the implement-backoff-requeue-machanism branch February 12, 2024 21:45
@tenzen-y tenzen-y mentioned this pull request Feb 13, 2024
Merged
@alculquicondor
Copy link
Contributor

/release-note-edit

WaitForPodsReady: Support a backoff re-queueing mechanism with configurable limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@denkensk denkensk Awaiting requested review from denkensk

@trasc trasc Awaiting requested review from trasc

@mimowo mimowo Awaiting requested review from mimowo

@alculquicondor alculquicondor Awaiting requested review from alculquicondor

Assignees

@alculquicondor alculquicondor

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API kind/feature Categorizes issue or PR as related to a new feature. lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Milestone
v0.6
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@tenzen-y @k8s-ci-robot @alculquicondor