Unschedulable Pod might take a long time to get the condition set

[alculquicondor]

What happened?

If there is a connection failure when updating the Pod status, we don't retry.
Furthermore, due to #108761, we won't have another chance until much later.

The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods.

What did you expect to happen?

We should have stronger guarantees to mark a Pod as Unschedulable.

How can we reproduce it (as minimally and precisely as possible)?

You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver.

Anything else we need to know?

No response

Kubernetes version

master

This is particularly problematic in 1.24, due to #108761.

Cloud provider

Any

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

@alculquicondor: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

/sig scheduling

[ahg-g]

That is bad, retrying in place on transient errors should probably be ok.

[sanposhiho]

I'll address it.
/assign

[alculquicondor]

Thanks!