add a new filter goaway which could send GOAWAY probabilistically to help balance HTTP2 requests

[answer1991]

What type of PR is this?
/kind feature

What this PR does / why we need it:

API Server and client use HTTP2 to connect each other, which means client always hit the same API Server across different request and the requests are NOT balanced even if we use a load-balance to expose API Server.

With this PR merged, a new filter goaway could send GOAWAY to HTTP2 client probabilistically. After client received GOAWAY, the in-flight watch request will NOT be influenced, and the new requests will use a new TCP connection, which means load-balance will work after client receive GOAWAY

Which issue(s) this PR fixes:

None

Special notes for your reviewer:

Nothing yet.

Does this PR introduce a user-facing change?:

Apiserver add a new flag --goaway-chance which is the fraction of requests that will be closed gracefully(GOAWAY) to prevent HTTP/2 clients from getting stuck on a single apiserver. 
After the connection closed(received GOAWAY), the client's other in-flight requests won't be affected, and the client will reconnect. 
The flag min value is 0 (off), max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
Clusters with single apiservers, or which don't use a load balancer, should NOT enable this.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

None

[k8s-ci-robot]

Hi @answer1991. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dixudx]

/ok-to-test

[answer1991]

/hold

let me do some more test firstly

[answer1991]

/hold cancel

The testing result is awesome. After client received GOAWAY, the in-flight watch request will NOT be influenced, and the new requests will use a new TCP connection, which means load-balance will work after client receive GOAWAY. Please see details and result in my test cases.

[answer1991]

cc @lavalamp @wojtek-t @deads2k

[dixudx]

/lgtm

[lavalamp]

Can you edit the release note to speak about the flag rather than the filter? Thanks.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: answer1991, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• staging/src/k8s.io/apiserver/OWNERS [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[answer1991]

updated release note.

/hold cancel

[answer1991]

Merge progress seems blocked, let me fix the validating error message nit and push again to trigger the merge progress

[answer1991]

@lavalamp kindly ping for a LGTM label. I removed the /hold label seems to trigger an authorize issue.

BTW, fix the validating error message nit.

Can we do an e2e run with this turned on before we merge

Not sure how to config the e2e API Server flags.

[deads2k]

/lgtm

[pidb]

Can I set it up for different resources?

[answer1991]

Can I set it up for different resources?

It's a HTTP filter and not care about resources.