Pod status updates take longer to propagate to the API than necessary

[smarterclayton]

What happened?

#107897 originally identified a number of inefficiencies in how pod status is reported back to the API:

1. Pods are queued in a channel and then processed one by one - this causes HoL blocking and writes every observed status change, but pod status is level driven (split into kubelet: Remove status manager channel #116615 for inclusion into Give terminal phase correctly to all pods that will not be restarted #115331)
2. Status is a GET then PATCH, but we now have server side apply and using it would reduce 2 RT to 1 RT and reduce QPS
3. We treat all status updates the same, when in reality certain transitions should be prioritized (unready -> ready, Running -> Succeeded|Failed)
4. Status is single threaded, and we could easily try multiple
5. It's hard to know when our status cache has been brought up to date (so that we can bypass a SSA attempt), we could use ResourceVersion tracking for that (apimachinery proposed allowing it to be parsed)

In general these contribute to #113606 and should be improved.

/sig node
/sig scalability

What did you expect to happen?

Faster

How can we reproduce it (as minimally and precisely as possible)?

See kubelet logs. Several e2e tests stress this flow, but we need a test that shows the improvements.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
# paste output here

Cloud provider

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[sftim]

We treat all status updates the same, when in reality certain transitions should be prioritized (unready -> ready, Running -> Succeeded|Failed)

One key actual state report for me is to see that the kubelet has at least seen the pod and, if you like, acknowledged its responsibility for further status updates. I think that's the update to set status.startTime.

[smarterclayton]

Right, that's another good one. Please add others. Note however that it may take 50ms for a round trip of a pod status, and some containers might start and go ready within that time. Would be good to think through how we want to bias towards these, or whether workload characteristics might matter.

[pacoxu]

/sig node
/sig scalability

[sftim]

With SSA, the kubelet can fire off multiple PATCH requests that - the kubelet believes - can arrive in an arbitrary order at the API server and produce the right outcome. I mean that's the dream, right?

[smarterclayton]

Right - Kubelet would simply need to guarantee that no two SSAs for the same pod are inflight at once.

[sftim]

kubelet would simply need to guarantee that no two SSAs for the same pod are inflight at once.

Not even that. No two conflicting SSAs. For example, I think we can send a request to apply status.startTime and, regardless of its outcome, follow that up with a request that applies status.startTime (to the same value!) and also updates status.conditions. The order that these apply shouldn't matter.

[SergeyKanzhelev]

/triage accepted

[smarterclayton]

Re: two inflight SSAs for non-conflicting operations to the same pod

For our own sanity, we probably don't want to have to reason about that while debugging :)

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted