Allow for recreation of pods once fully terminated in the job controller

[kannon92]

Enhancement Description

• One-line enhancement description (can be used as a release note): For Job controllers we want to optionally include a way for users to include terminating pods in the active pod count.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3939-allow-replacement-when-fully-terminated

• Discussion Link: Jobs create replacement Pods as soon as a Pod is marked for deletion kubernetes#115844

• Primary contact (assignee): @kannon92, @dejanzele

• Responsible SIGs: sig-apps

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.28
  • Beta release target (x.y): 1.29
  • Beta terminating fixes release target (x.y): 1.31
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): KEP-3939: Consider terminating pods in job controller #3940
  • Code (k/k) update PR(s):
    • Job API - add podReplacementPolicy and terminating field to job api kubernetes#119301
    • Job Controller Changes - Job: create replacement pods only after terminated kubernetes#117015
    • PodGC - KEP-3939: pod gc changes for pod replacement policy kep kubernetes#118772
  • Docs (k/website) update PR(s):
    • Job Section - Add information about PodReplacementPolicy in Job API website#41745
    • Alpha Blog Post - Blog post on new Job features in 1.28 (PodReplacementPolicy and BackoffLimitPerIndex) website#41924

• Beta

  • KEP (k/enhancements) update PR(s): KEP-3939: promote to beta in 1.29. #4252
  • Code (k/k) update PR(s):
    • Add a missing integration test for PodReplacementPolicy kubernetes#119912
    • Fix panic when enablement of pod replacement policy is skewed kubernetes#121011
    • Remove terminating count from rmAtLeast kubernetes#121147
    • cleanup: improve assertions for Failed PodReplacementPolicy integration test cases kubernetes#121274
    • Fix tracking of terminating Pods when nothing else changes kubernetes#121342
    • Metrics addition: feat: add job_pods_creation_total metric kubernetes#121481
    • Feature Enable and E2E Tests: Switch feature flag to beta for pod replacement policy and add e2e test kubernetes#121491
  • Docs (k/website) update(s):
    • Graduate JobPodReplacementPolicy to beta website#43529

• Beta (1.31)

  • KEP (k/enhancements) update PR(s): Update the managedBy KEP with fix for terminating pods #4657
  • Code (k/k) update PR(s):
    • Count terminating pods when deleting active pods for failed jobs kubernetes#125175
    • Delay setting terminal Job conditions until all pods are terminal kubernetes#125510
  • Docs (k/website) update(s):
    • Update the docs for JobManagedBy and JobPodReplacementPolicy related to pod termination website#46808

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kannon92]

/sig apps

[wangyysde]

/cc @wangyysde

[alculquicondor]

/assign
I'm taking over the KEP while @kannon92 is on PTO

[soltysh]

/stage alpha
/milestone v1.28
/label lead-opted-in

[aramase]

Hello @alculquicondor 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 01:00 UTC Friday, 16th June 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the latest template has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.28
• KEP readme has a updated detailed test plan section filled out
• KEP readme has up to date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[Atharva-Shinde]

Hey @alculquicondor @kannon92
With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you :)

[kannon92]

Thank you so much to @alculquicondor for taking this over while I was on PTO. It is so exciting to see that this is tracked for this release. @alculquicondor reached out and I will take over the implementation of this PR.

/assign

[aramase]

Hey again @alculquicondor @kannon92 👋

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

Here’s the enhancement’s state for the upcoming code freeze:

• All the PRs that are related to your enhancement are linked in the above issue description (for tracking purposes). This includes code, tests, and documentation related PR/s.
• All code related PR/s are merged or are in merge-ready state ( i.e they have approved and lgtm labels applied) by the code freeze deadline. This includes any tests related PR/s too.
  • add podReplacementPolicy and terminating field to job api kubernetes#119301
  • Job: create replacement pods only after terminated kubernetes#117015
  • KEP-3939: pod gc changes for pod replacement policy kep kubernetes#118772
  • Add information about PodReplacementPolicy in Job API website#41745

If there are any other k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[Atharva-Shinde]

Thanks for filling an exception request! I'll reapply the v1.28 milestone if the exception request is approved.

[dejanzele]

@sreeram-venkitesh, In my opinion, we merged everything which is required for PodReplacementPolicy to go into Beta and the requirements are done.

[kannon92]

k/website is not quite done but code freeze wise, we are looking good.

[kannon92]

I updated this checklist. I think we have each component necessary for 1.29 now and we are good to go for 1.29. Thank you @dejanzele for leading this effort. And thank you to @alculquicondor for doing the necessary docs updates!

[salehsedghpour]

/remove-label lead-opted-in

[salehsedghpour]

Hello 👋 1.30 Enhancements Lead here,

I'm closing milestone 1.29 now,
If you wish to progress this enhancement in v1.30, please follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board and finally add /milestone v1.30. Thanks!

/milestone clear

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[atiratree]

#4368 (#4657) managed-by feature changes the handling of terminating pods. It is better to wait for that one, before graduating PodReplacementPolicy to GA.

[atiratree]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kannon92]

/remove-lifecycle stale

[mimowo]

@kannon92 please add the following PRs to the list in the description (for Beta in 1.31):

• (KEP): Update the managedBy KEP with fix for terminating pods #4657
• (code): Count terminating pods when deleting active pods for failed jobs kubernetes#125175 & Delay setting terminal Job conditions until all pods are terminal kubernetes#125510
• docs: Update the docs for JobManagedBy and JobPodReplacementPolicy related to pod termination website#46808

[kannon92]

@mimowo done.

The code you linked too is the same PR. Was there another one you wanted to include?

[mimowo]

Yeah, I meant to add kubernetes/kubernetes#125510 (updated the message above)

[kannon92]

Thanks for bringing that up. They are all updated now.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[kannon92]

/remove-lifecycle stale

[kannon92]

@mimowo should we promote to stable in 1.33?