Retriable and non-retriable Pod failures for Jobs

[alculquicondor]

Enhancement Description

• One-line enhancement description (can be used as a release note): An API to influence retries based on exit codes and/or pod deletion reasons.

• Kubernetes Enhancement Proposal: https://git.k8s.io/enhancements/keps/sig-apps/3329-retriable-and-non-retriable-failures

• Discussion Link: RFE: ability to define special exit code to terminate existing job kubernetes#17244

• Primary contact (assignee): @alculquicondor

• Responsible SIGs: apps, api-machinery, scheduling

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.25
  • Beta release target (x.y): 1.26
  • Stable release target (x.y): 1.31

• Alpha

  • KEP (k/enhancements) update PR(s):
    • KEP-3329 Add KEP for Retriable and non-retriable Pod failures for Jobs #3374
    • Update KEP-3329 "Retriable and non-retriable Pod failures for Jobs" #3438
    • Additional update for KEP-3329 "Retriable and non-retriable Pod failures for Jobs" #3447
    • Updates to KEP-3329 "Retriable and non-retriable Pod failures for Jobs" #3452
  • Code (k/k) update PR(s):
    • Refactor gc_controller to do not use the deletePod stub kubernetes#111070
    • Refactor taint_manager to do not use getPod and getNode stubs kubernetes#111084
    • Add integration test for podgc kubernetes#111091
    • Append new pod conditions when deleting pods to indicate the reason for pod deletion kubernetes#110959
    • Support handling of pod failures with respect to the configured rules kubernetes#111113
    • Add worker to clean up stale DisruptionTarget condition kubernetes#111475
  • Docs (k/website) update PR(s): Add docs for KEP-3329 Retriable and non-retriable Pod failures for Jobs website#35219

• Beta

  • KEP (k/enhancements) update PR(s):
    • Update KEP-3329 "Retriable and non-retriable Pod failures for Jobs" for Beta #3463
    • Update for "Retriable and non-retriable Pod failures for Jobs" #3646
    • Testgrid links to e2e tests for "KEP-3329: Retriable and non-retriable Pod failures for Jobs" #3769
    • Update for second Beta with GA criteria for "KEP-3329: Retriable and non-retriable Pod failures for Jobs" #3757
    • v1.28
      • Beta update for Kubelet preemption (1.28) "KEP-3329: Retriable and non-retriable Pod failures for Jobs"  #3965
      • KEP-3939: Consider terminating pods in job controller #3940
    • v1.30
      • KEP-3329 pod failure policy update for 1.30 #4442
  • Code (k/k) update PR(s):
    • Add pod disruption conditions for kubelet-initiated failures kubernetes#112360
    • Extend metrics with the new labels kubernetes#113324
    • Use SSA to add pod failure conditions kubernetes#113304
    • Enable the "Retriable and non-retriable pod failures for jobs" feature into beta kubernetes#113360
    • Add e2e test for job pod failure policy used to match pod disruption kubernetes#113812
    • Fix disruption controller permissions to allow patching pod's status kubernetes#113580
    • Fix match onExitCodes when Pod is not terminated kubernetes#113856
    • Wait for Pods to finish before considering Failed in Job kubernetes#113860
    • Add e2e test to ignore failures with 137 exit code kubernetes#113927
    • Fix clearing of rate-limiter for the queue of checks for cleaning stale pod disruption conditions kubernetes#114770
    • Adjust DisruptionTarget condition message to do not include preemptor pod metadata kubernetes#114914
    • PodGC should not add DisruptionTarget condition for pods which are in terminal phase kubernetes#115056
    • Give terminal phase correctly to all pods that will not be restarted kubernetes#115331
    • API-initiated eviction: handle deleteOptions correctly kubernetes#116554
    • Add DisruptionTarget condition when preempting for critical pod kubernetes#117586
    • Job: create replacement pods only after terminated kubernetes#117015
    • Use Patch instead of SSA for Pod Disruption condition kubernetes#121103
  • Docs (k/website) update(s):
    • Promote "Retriable and non-retriable pod failures for Jobs" to Beta website#37242
    • Document for "Wait for Pods to finish before considering Failed in Job" website#38040
    • Extend documentation on PodGC focusing on PodDisruptionConditions enabled website#38042
    • Update docs for KEP3329: "Retriable and non-retriable Pod failures for jobs website#39809
    • Add information about PodReplacementPolicy in Job API website#41745

• Stable

  • KEP (k/enhancements) update PR(s): Graduate Job Pod Failure Policy to stable #4661
  • Code (k/k) update PR(s):
    • scheduler: Test that the DisruptionTarget condition is added at preemption time kubernetes#125533
    • Graduate JobPodFailurePolicy to stable kubernetes#125442
    • Graduate PodDisruptionConditions to stable kubernetes#125461
    • Promote JobPodFailurePolicy and PodDisruptionConditions e2e tests to Conformance kubernetes#125482
    • Use omitempty for optional fields in Job Pod Failure Policy kubernetes#126046
    • Fix a scheduler preemption issue where the victim isn't properly patched, leading to preemption not functioning as expected kubernetes#126644
    • clean up codes after PodDisruptionConditions was promoted to GA  kubernetes#125994
    • cleanup after JobPodFailurePolicy is promoted to GA kubernetes#126102
  • Docs (k/website) update(s):
    • Graduate Job Pod Failure Policy to stable website#46807
    • Add note that scenarios for DisruptionTarget must be retriable website#47378
    • blogpost: Pod Failure Policy for Jobs Goes GA website#46896

[alculquicondor]

/sig apps
/wg batch

[alculquicondor]

/assign

[mimowo]

/assign

[alculquicondor]

/sig scheduling
/sig api-machinery

[Priyankasaggu11929]

Hello @alculquicondor 👋, 1.25 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PT on Thursday June 23, 2022, which is just over 2 days from now.

For note, This enhancement is targeting for stage alpha for 1.25 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP file using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable
• KEP has a updated detailed test plan section filled out
• KEP has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The open PR #3374 is addressing all the listed criteria above. We would just require getting it merged by the Enhancements Freeze.

For note, the status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[Priyankasaggu11929]

With KEP PR #3374 merged, the enhancement is ready for the 1.25 Enhancements Freeze.

For note, the status is now marked as tracked. Thank you so much!

[kcmartin]

Hello @alculquicondor 👋, 1.25 Release Docs Lead here.
This enhancement is marked as ‘Needs Docs’ for 1.25 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.25 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by August 4.
 Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

[Atharva-Shinde]

Hi @alculquicondor @mimowo, Enhancements team here again 👋

Checking in as we approach Code Freeze at 01:00 UTC on Wednesday, 3rd August 2022.

Please ensure that the following items are completed before the code-freeze:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
  Refactor gc_controller to do not use the deletePod stub kubernetes#111070
  Refactor taint_manager to do not use getPod and getNode stubs kubernetes#111084
  Add integration test for podgc kubernetes#111091
  Append new pod conditions when deleting pods to indicate the reason for pod deletion kubernetes#110959
  Support handling of pod failures with respect to the configured rules kubernetes#111113
  Add worker to clean up stale DisruptionTarget condition kubernetes#111475
• All PRs are fully merged by the code freeze deadline.
  Append new pod conditions when deleting pods to indicate the reason for pod deletion kubernetes#110959
  Support handling of pod failures with respect to the configured rules kubernetes#111113
  Add worker to clean up stale DisruptionTarget condition kubernetes#111475

Let me know if there are any additional k/k PRs besides the ones listed above

Currently, the status of the enhancement is marked as at-risk

Thanks :)

[mimowo]

@Atharva-Shinde @alculquicondor there is one more PR that should be included before the code freeze: kubernetes/kubernetes#111475

[edithturn]

Hello, @alculquicondor and @mimowo!

👋 from the v1.31 Communications Team!

We'd love for you to opt in to write a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

Hello, @alculquicondor and @mimowo!

Just a friendly reminder: if you need to open a Feature Blog placeholder PR against the website repository, please go ahead and do so. If you have any questions or need any help, feel free to reach out! For more information about writing a blog, check out the blog contribution guidelines.

Remember, the deadline for this is coming up soon: Wednesday, July 03, 2024, at 18:00 PDT.

Thank you!!
Edith

[tjons]

Hey again @mimowo - 👋 Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024. .

Here's where this enhancement currently stands:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

For this enhancement, it looks like the following PRs are open and need to be merged before code freeze:

• Promote JobPodFailurePolicy and PodDisruptionConditions e2e tests to Conformance kubernetes#125482
• Graduate PodDisruptionConditions to stable kubernetes#125461
• Graduate JobPodFailurePolicy to stable kubernetes#125442

If you anticipate missing code freeze, you can file an exception request in advance.

Also, please let me know if there are other PRs in k/k we should be tracking for this KEP.
As always, we are here to help if any questions come up. Thanks!

[tjons]

With all PRs to k/k merged, this KEP is now tracked for code freeze!

[sreeram-venkitesh]

@mimowo Look like kubernetes/kubernetes#126169 has tests related to this KEP. Please make sure to get it merged before the test freeze deadline (01:00 UTC Wednesday 31st July 2024 / 19:00 PDT Tuesday 30th July 2024).

[mimowo]

@mimowo Look like kubernetes/kubernetes#126169 has tests related to this KEP. Please make sure to get it merged before the test freeze deadline (01:00 UTC Wednesday 31st July 2024 / 19:00 PDT Tuesday 30th July 2024).

@sreeram-venkitesh thanks for reaching out, I have updated the PR and will try to merge it, but OTOH I don't think it is required for this release cycle, because we will not promote these tests in this cycle anyway (per kubernetes/kubernetes#125482 (comment)). We have promoted already 2 tests which didn't have the flakiness issue: kubernetes/kubernetes#125482

[mimowo]

Just a friendly reminder: if you need to open a Feature Blog placeholder PR against the website repository, please go ahead and do so. If you have any questions or need any help, feel free to reach out! For more information about writing a blog, check out the blog contribution guidelines.

@edithturn I was busy preparing the code updates for the release cycle and missed this message, but I have already opened the placeholder PR for the blog-post, see #3329 (comment), and I'm working on the content. Can we still include it?

[kannon92]

@mimowo @alculquicondor Can we close this issue since the feature is stable?

[alculquicondor]

There are some post GA tasks to complete https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3329-retriable-and-non-retriable-failures#deprecation

[tjons]

@alculquicondor in preparation for the next release, could you give me an ETA for when you expect the post GA tasks to complete? Do we need to track this KEP in the next cycle?

[mimowo]

The only pending work is to remove the feature-gate in 1.33.

I think we can close the issue, I see that other issues corresponding to GA features which are pending feature-gate removal are already closed, examples: AdmissionWebhookMatchConditions, AggregatedDiscoveryEndpoint, APIListChunking.

[mimowo]

It might be confusing that we have the "Modify the code to ignore the PodDisruptionConditions and JobPodFailurePolicy feature gates" to reflect the actual state" task in the Deprecation, but it was already done. I have created the KEP update: #4835. @alculquicondor please add these two PRs to the implementation list in the issue: kubernetes/kubernetes#125994 and kubernetes/kubernetes#126102

[alculquicondor]

Added

[alculquicondor]

Closing the issue seems ok, given the precedent of other KEPs that already graduated.

[mimowo]

/close

[k8s-ci-robot]

@mimowo: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.