Revert "Revert "Merge pull request #107797 from tkashem/revert-107456""

[liggitt]

This reverts commit b0b4609.

Reverts #108383 (comment)
Fixes #109026

Integration tests have been very flaky since merge, with 47k panics and 3gb logs per run.

This should not be re-added until #109026 is resolved and there's positive confirmation we do not have panics in the integration tests

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/pkg/util/flowcontrol/OWNERS [liggitt]
• test/integration/apiserver/OWNERS [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

cc @wojtek-t @tkashem

[tkashem]

/lgtm

/cc @MikeSpreitzer
Thanks @liggitt

[dims]

/retest

[liggitt]

there were two distinct bumps in test-integration failures (other than the go1.18 solid failure on CSR validation)

1. a few days ago, when Revert the revert of #107456 - "apf: change controller to use SSA for patches" #108383 merged
2. ~yesterday, cause still unknown / being investigated

so I think we have stacked failures, and this resolves one of them

[liggitt]

/retest

[tkashem]

@liggitt @aojea from a failed integration test, i see etcd client retry interceptor is at attempt: 95

{"level":"warn","ts":"2022-03-25T19:04:08.449Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc020abf340/127.0.0.1:2379","attempt":95,"error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}

have we recently changed any retry configuration for the etcd retry interceptor? this seems to be an excessive number of retries.

this seems to trigger the retry from the storage layer, many of the below errors

I0325 19:04:09.061551  121727 store.go:451] GuaranteedUpdate of /713a0552-dbd2-41a3-a59f-7bb114e8dea2/secrets/status-code/test failed because of a conflict, going to retry

[tkashem]

/test pull-kubernetes-integration

[liggitt]

@liggitt @aojea from a failed integration test, i see etcd client retry interceptor is at attempt: 95

grpc client got bumped from 1.40 to 1.44 this morning as a side effect of #108954

[tkashem]

it may not be the root cause of the integration test timeout, but i think we should have some sort of bound to the retry. Naturally it should be bound to the associated request context, but not all requests have a bound deadline. Maybe we should set a maximum of retry attempt to a lower value like 10 or something?

[liggitt]

opened #109038 for the acute integration failures seen starting today

[liggitt]

Maybe we should set a maximum of retry attempt to a lower value like 10 or something?

let's discuss in #109038

[aojea]

/retest

k8s.io/kubernetes/test/integration/apiserver: TestPatchConflicts expand_more	3m30s
k8s.io/kubernetes/test/integration/apiserver/apply: TestScaleAllResources expand_more	0s
k8s.io/kubernetes/test/integration/auth: TestDiscoveryUpgradeBootstrapping expand_more	0s
k8s.io/kubernetes/test/integration/deployment: TestDeploymentRollingUpdate expand_more
```

[aojea]

k8s.io/kubernetes/pkg/kubelet: TestUpdatePodWithTerminatedPod expand_less

/retest