Mutable Container Resources when Job is suspended

[kannon92]

Enhancement Description

• One-line enhancement description (can be used as a release note): Allow for mutating container requests/limits on a PodTemplate when job is suspended

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/5440-mutable-job-pod-resource-updates

• Discussion Link: [InPlacePodVerticalScaling] Relaxing immutability of Pod Templates resources/limits kubernetes#132436

• PRs by stage and milestone:

  • Alpha - v1.35
    • KEP (k/enhancements) update PR(s):
      • [KEP-5440]: Mutable container resources on PodTemplates for suspended jobs #5441
    • Code (k/k) update PR(s):
      • [KEP-5440]: MutablePodResourcesForSuspendedJobs  kubernetes#132441
    • Docs (k/website) update PR(s):
      • [KEP-5440] Add documentation for mutable pod resources while job is suspended website#52800

• Beta - v1.36

  • KEP (k/enhancements) update PR(s): KEP-5440: move to beta #5703
  • Code (k/k) update PR(s):
    • [KEP-5440]: Add integration test for MutablePodResourcesForSuspendedJobs with Pod Replacement Policy = Failed kubernetes#135381
    • set KEP-5440 to enabled by default kubernetes#135965
  • Docs (k/website) update(s):

• Stable - v1.xx

  • KEP (k/enhancements) update PR(s):
  • Code (k/k) update PR(s):
  • Docs (k/website) update(s):
    -->

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[kannon92]

/sig apps

[kannon92]

So I have a POC and an enhancement with this for device plugins and classic resources (cpu/memory).

I realize that more work is needed for DRA.

Right now, DRA does not allow modification of a ResourceClaimTemplate once it is created.

And you are also not able to change the claim on a PodTemplate.

I am not sure the path forward for DRA though.

Option 1:

• Someone can create a new ResourceClaimTemplate. Patch workload to point to new ResourceClaimTemplate.
• This is at least audit-able from api logs as user must patch workload.

Option 2:

• Someone modifies ResourceClaimTemplate.
• Workload would reflect updated ResourceClaimTemplate.
• Workload author can't really control if someone changes this under them?

[kannon92]

I met with wg-device-management about DRA.

For now, one solution if there are no pods in use, is to recreate the resourceClaimTemplate with update requirements.

With this we don't need to relax immutability for claims.

[kannon92]

@soltysh @janetkuo I would like to try and get this in for 1.35 if possible.

[soltysh]

/label lead-opted-in
/milestone v1.35
/stage alpha