[release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default #253

slashpai · 2023-11-03T06:13:07Z

cherry-picked from upstream prometheus-operator

This change mitigates CVE-2023-44487 by disabling HTTP2 by default and forcing HTTP/1.1 until the Go standard library and golang.org/x/net are fully fixed. Right now, it is possible for authenticated and unauthenticated users to hold open HTTP2 connections and consume huge amounts of memory. It is possible to revert back the change by using the `--web.enable-http2` argument. Before this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted h2 [...] * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /metrics] * h2h3 [:scheme: https] * h2h3 [:authority: localhost:8443] * h2h3 [user-agent: curl/8.0.1] * h2h3 [accept: */*] * Using Stream ID: 1 (easy handle 0x5594d4614b10) [...] > GET /metrics HTTP/2 [...] ``` After this change: ``` curl -kv https://localhost:8443/metrics * Trying 127.0.0.1:8443... * Connected to localhost (127.0.0.1) port 8443 (#0) * ALPN: offers h2,http/1.1 [...] * ALPN: server accepted http/1.1 [...] * using HTTP/1.1 > GET /metrics HTTP/1.1 > Host: localhost:8443 > User-Agent: curl/8.0.1 > Accept: */* [...] < HTTP/1.1 200 OK [...] ``` See also: * kubernetes/kubernetes#121120 * kubernetes/kubernetes#121197 * golang/go#63417 (comment) Signed-off-by: Simon Pasquier <spasquie@redhat.com> (cherry picked from commit a62e814)

openshift-ci-robot · 2023-11-03T06:13:12Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is invalid:

expected dependent Jira Issue OCPBUGS-21637 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

cherry-picked from upstream prometheus-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

simonpasquier

/lgtm

simonpasquier · 2023-11-03T08:13:06Z

/label backport-risk-assessed

slashpai · 2023-11-03T12:07:34Z

/Jira refresh

openshift-ci-robot · 2023-11-03T12:07:38Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is invalid:

expected dependent Jira Issue OCPBUGS-21637 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

/Jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

slashpai · 2023-11-04T04:29:13Z

/Jira refresh

openshift-ci-robot · 2023-11-04T04:29:17Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is invalid:

expected dependent Jira Issue OCPBUGS-21637 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/Jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

slashpai · 2023-11-05T05:30:55Z

/Jira refresh

openshift-ci-robot · 2023-11-05T05:30:59Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is invalid:

expected dependent Jira Issue OCPBUGS-21637 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/Jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

slashpai · 2023-11-06T06:00:52Z

/jira refresh

openshift-ci-robot · 2023-11-06T06:01:00Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.z) matches configured target version for branch (4.14.z)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-21637 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-21637 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
bug has dependents

Requesting review from QA contact:
/cc @juzhao

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

slashpai · 2023-11-06T06:29:37Z

/hold
Till OCPBUGS-22946 is evaluated

juzhao · 2023-11-06T07:49:16Z

/label cherry-pick-approved
/label qe-approved

openshift-ci-robot · 2023-11-06T07:49:27Z

@slashpai: This pull request references Jira Issue OCPBUGS-20881, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.14.z) matches configured target version for branch (4.14.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
dependent bug Jira Issue OCPBUGS-21637 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
dependent Jira Issue OCPBUGS-21637 targets the "4.15.0" version, which is one of the valid target versions: 4.15.0
bug has dependents

Requesting review from QA contact:
/cc @juzhao

In response to this:

cherry-picked from upstream prometheus-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

After disabling HTTP2 connections by default in prometheus-operator#6028 started seeing verbose logging in admission webhook pods like: `ts=2023-11-06T01:50:34.601774794Z caller=stdlib.go:105 caller=server.go:3215 msg="http: superfluous response.WriteHeader call from main.newSrv.func1 (main.go:173)" ` Go doc says If WriteHeader is not called explicitly, the first call to Write will trigger an implicit WriteHeader(http.StatusOK). This fix is to change the order of the method invocation so log doesn't complain about this Signed-off-by: Jayapriya Pai <slashpai9@gmail.com> (cherry picked from commit 374f4b3)

simonpasquier

/lgtm

slashpai · 2023-11-06T16:06:57Z

/hold cancel
verified on 4.15 cluster fix is working

openshift-ci · 2023-11-06T16:07:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: raptorsun, simonpasquier, slashpai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [raptorsun,simonpasquier,slashpai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2023-11-06T19:17:07Z

@slashpai: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2023-11-06T19:20:28Z

@slashpai: Jira Issue OCPBUGS-20881: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-20881 has been moved to the MODIFIED state.

In response to this:

cherry-picked from upstream prometheus-operator

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-merge-robot · 2023-11-07T04:16:06Z

Fix included in accepted release 4.14.0-0.nightly-2023-11-06-203803

openshift-bot · 2023-11-22T03:11:51Z

[ART PR BUILD NOTIFIER]

This PR has been included in build prometheus-operator-container-v4.14.0-202311211133.p0.g73729ed.assembly.stream for distgit prometheus-operator.
All builds following this will include this PR.

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 3, 2023

openshift-ci bot requested review from raptorsun and simonpasquier November 3, 2023 06:13

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 3, 2023

slashpai changed the title ~~OCPBUGS-20881: fix: disable HTTP2 connections by default~~ [release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default Nov 3, 2023

simonpasquier reviewed Nov 3, 2023

View reviewed changes

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 3, 2023

openshift-ci bot assigned simonpasquier Nov 3, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 3, 2023

raptorsun approved these changes Nov 3, 2023

View reviewed changes

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Nov 6, 2023

openshift-ci bot requested a review from juzhao November 6, 2023 06:01

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 6, 2023

openshift-ci bot added cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. qe-approved Signifies that QE has signed off on this PR labels Nov 6, 2023

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Nov 6, 2023

simonpasquier reviewed Nov 6, 2023

View reviewed changes

openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Nov 6, 2023

openshift-merge-bot bot merged commit 73729ed into openshift:release-4.14 Nov 6, 2023
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default #253

[release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default #253

slashpai commented Nov 3, 2023

openshift-ci-robot commented Nov 3, 2023

simonpasquier left a comment

simonpasquier commented Nov 3, 2023

slashpai commented Nov 3, 2023

openshift-ci-robot commented Nov 3, 2023

slashpai commented Nov 4, 2023

openshift-ci-robot commented Nov 4, 2023

slashpai commented Nov 5, 2023

openshift-ci-robot commented Nov 5, 2023

slashpai commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

slashpai commented Nov 6, 2023

juzhao commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

simonpasquier left a comment

slashpai commented Nov 6, 2023

openshift-ci bot commented Nov 6, 2023

openshift-ci bot commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

openshift-merge-robot commented Nov 7, 2023

openshift-bot commented Nov 22, 2023

[release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default #253

[release-4.14] OCPBUGS-20881: fix: disable HTTP2 connections by default #253

Conversation

slashpai commented Nov 3, 2023

openshift-ci-robot commented Nov 3, 2023

simonpasquier left a comment

Choose a reason for hiding this comment

simonpasquier commented Nov 3, 2023

slashpai commented Nov 3, 2023

openshift-ci-robot commented Nov 3, 2023

slashpai commented Nov 4, 2023

openshift-ci-robot commented Nov 4, 2023

slashpai commented Nov 5, 2023

openshift-ci-robot commented Nov 5, 2023

slashpai commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

slashpai commented Nov 6, 2023

juzhao commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

simonpasquier left a comment

Choose a reason for hiding this comment

slashpai commented Nov 6, 2023

openshift-ci bot commented Nov 6, 2023

openshift-ci bot commented Nov 6, 2023

openshift-ci-robot commented Nov 6, 2023

openshift-merge-robot commented Nov 7, 2023

openshift-bot commented Nov 22, 2023