Kubernetes 1.28 Major Themes

[sanchita-07]

This is a discussion to discuss Major Themes for the v1.28 release.

Here is the discussion thread from the last release cycle.

cc @kubernetes/release-team-leads
cc @AnaMMedina21 @fsmunoz @michaelfromyeg @muddapu @rashansmith

[xing-yang]

From SIG-Storage, we have the following:

New Features:

• The non-graceful node shutdown feature moved to GA in v1.28. This feature allows stateful workloads to restart on a different node if the original node is shutdown unexpectedly or ends up in a non-recoverable state perhaps due to hardware failure or unresponsive OS.
• The Retroactive default StorageClass assignment feature moved to GA in v1.28. This feature makes it easier to change default StorageClass by allowing default storage class assignment to be retroactive for existing unbound persistent volume claims without any storage class assigned.

In-tree Driver Deprecation/Removals

• CSI Migration for GCE PD moved to GA in v1.25. The in-tree GCE PD plugin is targeted for removal in v1.28 (CSI migration complete).
• Ceph RBD in-tree Plugin has been deprecated in v1.28 and is targeted for removal in v1.31 (no CSI migration planned).
• Ceph FS in-tree Plugin has been deprecated in v1.28 and is targeted for removal in v1.31 (no CSI migration planned).

cc @saad-ali @msau42 @jsafrane

[carlory]

Hi @xing-yang, the VolumeAttributesClass feature is deferred to the next release 1.29

[klueska]

One from sig node is the addition of CDI support to the device plugin API:
kubernetes/enhancements#4009

CDI provides a standardized way of injecting complex devices into a container (i.e. devices that logically require more than just a single /dev node to be injected for them to work). This new feature enables plugin developers to utilize the CDIDevices field added to the CRI in 1.27 to pass CDI devices directly to CDI enabled runtimes (of which containerd and crio-o are in recent releases).

More info on CDI itself can be found here.

[liggitt]

From SIG-Arch: starting with 1.28 control planes / 1.25 nodes, the Kubernetes skew policy is expands supported control-plane / node skew to 3 versions. This enables annual minor version upgrades of nodes while staying on supported minor versions. More information about an annual node upgrade strategy in the design and the updated skew policy at https://kubernetes.io/releases/version-skew-policy/

[logicalhan]

From SIG-Instrumentation: we've graduated the static analysis extensions to the metrics stability framework, which now includes support for BETA level metrics and auto-generation of metric documentation. More information about the respective stability levels (and the metrics themselves) can be found here.

[bradmccoydev]

Hi do you have the KEP link for this please?

[logicalhan]

https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/3498-extending-stability

[alculquicondor]

From SIG-Apps / WG-Batch:
Based on user feedback, we are increasing the functionality of the Job API, to support a wider variety of batch use cases. Highlights include the Pod replacement policy and backoff limit per Index to independently control the failures of indexes independently.

[SergeyKanzhelev]

Is it late to add more?

On SIG Node:

• Sidecar containers built-in support are now in alpha: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/753-sidecar-containers
• Memory swap entered beta: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/README.md
• PodResources API is entered GA
• As many customer switching to cgroupv2, we implemented auto-detection of Container Runtime's cgroup driver to use in kubelet.
• Expanded testing:
  • Kubelet standalone mode tests are now periodics
  • ARM64 e2e test are now running as periodics
  • Multi-numa environment enabled to run e2e tests for various managers

[sanchita-07]

Is it late to add more?

It's fine. 👍

[fedebongio]

For SIG API Machinery, here is our update:

Here's a list of what merged:

• CRD Validation Rules (CEL, 2nd beta, aiming for GA in 1.29)
• ValidatingAdmissionPolicy (CEL, promoted to beta)
• Webhook MatchConditions (CEL, promoted to beta)
• UVIP (alpha)
• CRD Validation Ratcheting (alpha)
• Consistent reads from cache (alpha)

Spotlight goes to the CEL feature promotions.

Additionally, we'd like to get people to try out UVIP, but also would be very important that the community tries to get some milage on Consistent reads from cache

(thank you @jpbetz for compiling the list!)

[aravindhp]

For SIG Windows, here is our update:

• Cri-only stats (@mansikulkarni96)
  • This extends cAdvisor-less, CRI-full Container and Pod Stats to Windows nodes
• Windows-service-proxy (@Damans227)
  • Working example of building your own kube-proxy out of tree using KPNG

[bradmccoydev]

Hi, is there a link to the KEP for the Windows-service-proxy?

[aravindhp]

Given it is out of tree work I am not sure but @jayunit100 will be able to tell you.