Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mitigate http/2 attacks by authenticated clients #121197

Open
enj opened this issue Oct 12, 2023 · 9 comments
Open

Mitigate http/2 attacks by authenticated clients #121197

enj opened this issue Oct 12, 2023 · 9 comments
Labels
sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@enj
Copy link
Member

enj commented Oct 12, 2023

golang/net@b225e7c does not sufficiently protect us from a malicious http/2 client. #121120 handles unauthenticated clients. This issue tracks mitigations for authenticated clients.

pprof svg from a single client with a single connection attempting to DOS a Kube API server that has max streams set to 100 (memory usage grew to 5 GB in a few mins before it stabilized - with multiple connections the API server would have easily OOM'd):

mem

xref: golang/go#63417 (comment)

@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2023
@liggitt
Copy link
Member

liggitt commented Oct 12, 2023

/sig api-machinery

@k8s-ci-robot
Copy link
Contributor

@liggitt: The label(s) sig/networking cannot be applied, because the repository doesn't have them.

In response to this:

/sig api-machinery networking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Oct 12, 2023
@liggitt
Copy link
Member

liggitt commented Oct 12, 2023

/sig network

@k8s-ci-robot k8s-ci-robot added the sig/network Categorizes an issue or PR as relevant to SIG Network. label Oct 12, 2023
@liggitt liggitt removed the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Oct 12, 2023
@k8s-ci-robot k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Oct 12, 2023
@enj
Copy link
Member Author

enj commented Oct 12, 2023

/milestone v1.29

@k8s-ci-robot k8s-ci-robot added this to the v1.29 milestone Oct 12, 2023
@hailkomputer
Copy link
Member

Hello @enj , Bug triage team lead here. I want to check the status.
The code freeze is starting 01:00 UTC Wednesday 1st November 2023 / 18:00 PDT Tuesday 31st October 2023 (about two weeks from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged and issue’s are addressed on time.
As the issue is tagged for 1.29, is it planned for this release?

simonpasquier added a commit to simonpasquier/prometheus-operator that referenced this issue Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 and forcing HTTP/1.1
until the Go standard library and golang.org/x/net are fully fixed. Right now,
it is possible for authenticated and unauthenticated users to hold open HTTP2
connections and consume huge amounts of memory.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier added a commit to simonpasquier/prometheus-operator that referenced this issue Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier added a commit to simonpasquier/prometheus-operator that referenced this issue Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier added a commit to simonpasquier/cluster-monitoring-operator that referenced this issue Oct 19, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as
demonstrated in kubernetes/kubernetes#121197.

To prevent any risk with HTTP2, this change disables HTTP2 at the server
level. There's no expected performance penalty because the web server is
only used for scraping metrics.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
simonpasquier added a commit to simonpasquier/prometheus-operator that referenced this issue Oct 19, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
sadlerap added a commit to sadlerap/service-binding-runtime that referenced this issue Oct 19, 2023
It appears that mitigating the recent http2 vulnerabilities (see
CVE-2023-44487 and CVE-2023-39325) requires [more than just a library
update to golang.org/x/net][1].  Until better mitigations have been
developed, disable http2 in both the metrics and webhooks servers.

[1]: kubernetes/kubernetes#121197

Signed-off-by: Andy Sadler <ansadler@redhat.com>
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/cluster-monitoring-operator that referenced this issue Oct 20, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as
demonstrated in kubernetes/kubernetes#121197.

To prevent any risk with HTTP2, this change disables HTTP2 at the server
level. There's no expected performance penalty because the web server is
only used for scraping metrics.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
openshift-cherrypick-robot pushed a commit to openshift-cherrypick-robot/cluster-monitoring-operator that referenced this issue Oct 24, 2023
Commit b3b011b was not enough to prevent HTTP2 connections as
demonstrated in kubernetes/kubernetes#121197.

To prevent any risk with HTTP2, this change disables HTTP2 at the server
level. There's no expected performance penalty because the web server is
only used for scraping metrics.

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
@aojea
Copy link
Member

aojea commented Oct 24, 2023

/cc

@enj
Copy link
Member Author

enj commented Oct 24, 2023

I have some ideas around this, but I think those changes will be too large / risky for this late in the release cycle.

/milestone v1.30

@k8s-ci-robot k8s-ci-robot modified the milestones: v1.29, v1.30 Oct 24, 2023
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Oct 25, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Oct 25, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Oct 26, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Oct 26, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
@ReToCode
Copy link

@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach.

@enj
Copy link
Member Author

enj commented Oct 27, 2023

@enj would it be possible to share how you tested this? (privately is also possible). We are working on mitigations for https://github.com/knative and I was testing with, but would be interested to see your approach.

Sorry, I cannot share the exploit code. Your approach is an okay approximation. You should however assume that a more sophisticated attack would lead to higher resource consumption.

slashpai pushed a commit to slashpai/prometheus-operator that referenced this issue Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
(cherry picked from commit a62e814)
slashpai pushed a commit to slashpai/prometheus-operator that referenced this issue Nov 3, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
(cherry picked from commit a62e814)
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Nov 7, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
davidmogar added a commit to redhat-appstudio/release-service that referenced this issue Nov 7, 2023
This is required to mitigate the CVE described here:
kubernetes/kubernetes#121197

Signed-off-by: David Moreno García <damoreno@redhat.com>
sadlerap added a commit to servicebinding/runtime that referenced this issue Nov 7, 2023
* disable http2 for metrics and webhooks by default

It appears that mitigating the recent http2 vulnerabilities (see
CVE-2023-44487 and CVE-2023-39325) requires [more than just a library
update to golang.org/x/net][1].  Until better mitigations have been
developed, disable http2 in both the metrics and webhooks servers.

[1]: kubernetes/kubernetes#121197

Signed-off-by: Andy Sadler <ansadler@redhat.com>

* cleanup http2 disabling methods

Until better mitigations are in place, disable HTTP2 in all cases.
Don't leave an option in place to re-enable it.

Signed-off-by: Andy Sadler <ansadler@redhat.com>

* fix generated drift

Signed-off-by: Andy Sadler <ansadler@redhat.com>

---------

Signed-off-by: Andy Sadler <ansadler@redhat.com>
adinhodovic pushed a commit to adinhodovic/prometheus-operator that referenced this issue Nov 13, 2023
This change mitigates CVE-2023-44487 by disabling HTTP2 by default and
forcing HTTP/1.1 until the Go standard library and golang.org/x/net are
fully fixed. Right now, it is possible for authenticated and
unauthenticated users to hold open HTTP2 connections and consume huge
amounts of memory.

It is possible to revert back the change by using the
`--web.enable-http2` argument.

Before this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted h2
[...]
* using HTTP/2
* h2h3 [:method: GET]
* h2h3 [:path: /metrics]
* h2h3 [:scheme: https]
* h2h3 [:authority: localhost:8443]
* h2h3 [user-agent: curl/8.0.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x5594d4614b10)
[...]
> GET /metrics HTTP/2
[...]
```

After this change:

```
curl -kv https://localhost:8443/metrics
*   Trying 127.0.0.1:8443...
* Connected to localhost (127.0.0.1) port 8443 (#0)
* ALPN: offers h2,http/1.1
[...]
* ALPN: server accepted http/1.1
[...]
* using HTTP/1.1
> GET /metrics HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.0.1
> Accept: */*
[...]
< HTTP/1.1 200 OK
[...]
```

See also:
* kubernetes/kubernetes#121120
* kubernetes/kubernetes#121197
* golang/go#63417 (comment)

Signed-off-by: Simon Pasquier <spasquie@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/network Categorizes an issue or PR as relevant to SIG Network. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Status: Tracked
Development

No branches or pull requests

6 participants