Mitigate http/2 attacks by authenticated clients #121197
Labels
sig/api-machinery
Categorizes an issue or PR as relevant to SIG API Machinery.
sig/network
Categorizes an issue or PR as relevant to SIG Network.
triage/accepted
Indicates an issue or PR is ready to be actively worked on.
Milestone
Comments
k8s-ci-robot
added
needs-sig
|
/sig api-machinery |
|
@liggitt: The label(s) In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
sig/api-machinery
|
/sig network |
k8s-ci-robot
added
the
sig/network
liggitt
removed
the
triage/accepted
k8s-ci-robot
added
needs-triage
|
/milestone v1.29 |
|
Hello @enj , Bug triage team lead here. I want to check the status. |
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>
5 tasks
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier
added a commit
to simonpasquier/cluster-monitoring-operator
that referenced
this issue
Oct 19, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <spasquie@redhat.com>
2 tasks
simonpasquier
added a commit
to simonpasquier/prometheus-operator
that referenced
this issue
Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>
sadlerap
added a commit
to sadlerap/service-binding-runtime
that referenced
this issue
Oct 19, 2023
It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <ansadler@redhat.com>
1 task
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/cluster-monitoring-operator
that referenced
this issue
Oct 20, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <spasquie@redhat.com>
openshift-cherrypick-robot
pushed a commit
to openshift-cherrypick-robot/cluster-monitoring-operator
that referenced
this issue
Oct 24, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as demonstrated in kubernetes/kubernetes#121197. To prevent any risk with HTTP2, this change disables HTTP2 at the server level. There's no expected performance penalty because the web server is only used for scraping metrics. Signed-off-by: Simon Pasquier <spasquie@redhat.com>
|
/cc |
|
I have some ideas around this, but I think those changes will be too large / risky for this late in the release cycle. /milestone v1.30 |
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Oct 25, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Oct 25, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Oct 26, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Oct 26, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
|
@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach. |
This was referenced Oct 26, 2023
Sorry, I cannot share the exploit code. Your approach is an okay approximation. You should however assume that a more sophisticated attack would lead to higher resource consumption. |
slashpai
pushed a commit
to slashpai/prometheus-operator
that referenced
this issue
Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)
5 tasks
slashpai
pushed a commit
to slashpai/prometheus-operator
that referenced
this issue
Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Nov 7, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar
added a commit
to redhat-appstudio/release-service
that referenced
this issue
Nov 7, 2023
This is required to mitigate the CVE described here: kubernetes/kubernetes#121197 Signed-off-by: David Moreno García <damoreno@redhat.com>
sadlerap
added a commit
to servicebinding/runtime
that referenced
this issue
Nov 7, 2023
* disable http2 for metrics and webhooks by default It appears that mitigating the recent http2 vulnerabilities (see CVE-2023-44487 and CVE-2023-39325) requires [more than just a library update to golang.org/x/net][1]. Until better mitigations have been developed, disable http2 in both the metrics and webhooks servers. [1]: kubernetes/kubernetes#121197 Signed-off-by: Andy Sadler <ansadler@redhat.com> * cleanup http2 disabling methods Until better mitigations are in place, disable HTTP2 in all cases. Don't leave an option in place to re-enable it. Signed-off-by: Andy Sadler <ansadler@redhat.com> * fix generated drift Signed-off-by: Andy Sadler <ansadler@redhat.com> --------- Signed-off-by: Andy Sadler <ansadler@redhat.com>
adinhodovic
pushed a commit
to adinhodovic/prometheus-operator
that referenced
this issue
Nov 13, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
golang/net@b225e7c does not sufficiently protect us from a malicious http/2 client. #121120 handles unauthenticated clients. This issue tracks mitigations for authenticated clients.
pprof svg from a single client with a single connection attempting to DOS a Kube API server that has max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):
xref: golang/go#63417 (comment)
The text was updated successfully, but these errors were encountered: