New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automated cherry pick of #121196: Prevent rapid reset http2 DOS on API server #121201
Automated cherry pick of #121196: Prevent rapid reset http2 DOS on API server #121201
Conversation
This change fully addresses CVE-2023-44487 and CVE-2023-39325 for the API server when the client is unauthenticated. The changes to util/runtime are required because otherwise a large number of requests can get blocked on the time.Sleep calls. For unauthenticated clients (either via 401 or the anonymous user), we simply no longer allow such clients to hold open http2 connections. They can use http2, but with the performance of http1 (with keep-alive disabled). Since this change has the potential to cause issues, the UnauthenticatedHTTP2DOSMitigation feature gate can be disabled to remove this protection (it is enabled by default). For example, when the API server is fronted by an L7 load balancer that is set up to mitigate http2 attacks, unauthenticated clients could force disable connection reuse between the load balancer and the API server (many incoming connections could share the same backend connection). An API server that is on a private network may opt to disable this protection to prevent performance regressions for unauthenticated clients. For all other clients, we rely on the golang.org/x/net fix in golang/net@b225e7c That change is not sufficient to adequately protect against a motivated client - future changes to Kube and/or golang.org/x/net will be explored to address this gap. The Kube API server now uses a max stream of 100 instead of 250 (this matches the Go http2 client default). This lowers the abuse limit from 1000 to 400. Signed-off-by: Monis Khan <mok@microsoft.com>
This makes backports safer by not changing any default behavior. Signed-off-by: Monis Khan <mok@microsoft.com>
/lgtm |
LGTM label has been added. Git tree hash: 80d3122f09372059fe4a0940b6baa70185dc11a0
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: enj, liggitt The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
/cc @kubernetes/release-managers |
These occasionally flake on CI: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/121200/pull-kubernetes-unit-go-compatibility/1712589824344461312 === Failed === FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true/http/1.1 (0.19s) authentication_test.go:653: expect TCP connection: 1, actual: 2 --- FAIL: TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true/http/1.1 (0.19s) === FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true (0.23s) --- FAIL: TestUnauthenticatedHTTP2ClientConnectionClose/other_skip=true (0.23s) === FAIL: vendor/k8s.io/apiserver/pkg/endpoints/filters TestUnauthenticatedHTTP2ClientConnectionClose (2.30s) Signed-off-by: Monis Khan <mok@microsoft.com>
/lgtm |
LGTM label has been added. Git tree hash: 44c0fe48ed03d51bdf86df76abdbb1e1c2b60a14
|
/triage accepted |
…#121201-kubernetes#121228-upstream-release-1.24 Automated cherry pick of kubernetes#121201 kubernetes#121228 upstream release 1.24
Cherry pick of #121120 on release-1.25. (#121120 with the feature gate disabled by default)
#121120 #121196: Prevent rapid reset http2 DOS on API server
Prevent rapid reset http2 DOS on API server
This change fully addresses CVE-2023-44487 and CVE-2023-39325 for
the API server when the client is unauthenticated.
The changes to util/runtime are required because otherwise a large
number of requests can get blocked on the time.Sleep calls.
For unauthenticated clients (either via 401 or the anonymous user),
we simply no longer allow such clients to hold open http2
connections. They can use http2, but with the performance of http1
(with keep-alive disabled).
Since this change has the potential to cause issues, the
UnauthenticatedHTTP2DOSMitigation feature gate can be disabled to
remove this protection (it is enabled by default). For example,
when the API server is fronted by an L7 load balancer that is set up
to mitigate http2 attacks, unauthenticated clients could force
disable connection reuse between the load balancer and the API
server (many incoming connections could share the same backend
connection). An API server that is on a private network may opt to
disable this protection to prevent performance regressions for
unauthenticated clients.
For all other clients, we rely on the golang.org/x/net fix in
golang/net@b225e7c
That change is not sufficient to adequately protect against a
motivated client - future changes to Kube and/or golang.org/x/net
will be explored to address this gap.
The Kube API server now uses a max stream of 100 instead of 250
(this matches the Go http2 client default). This lowers the abuse
limit from 1000 to 400.
Signed-off-by: Monis Khan mok@microsoft.com
Disable UnauthenticatedHTTP2DOSMitigation by default
This makes backports safer by not changing any default behavior.
Signed-off-by: Monis Khan mok@microsoft.com