Bug 1826225: Support edge-terminated h2 connections #328

Miciah · 2021-08-26T19:40:50Z

Fix a couple of godoc typos

pkg/router/template/plugin_test.go (TestHandleTCPEndpoints):
pkg/router/template/router.go (createServiceUnitInternal): Fix typos in the godoc for these functions.

Use HTTP/2 if endpoint's AppProtocol is "h2" or "h2c"

Check the appProtocol field on ports in Kubernetes Endpoints objects. If the route is a reencrypt route with appProtocol "h2" or a non-TLS or edge-terminated route with appProtocol "h2c", configure HAProxy to use HTTP/2 for the endpoint.

Add checks in the dynamic config manager to force a full reload when a new endpoint specifies appProtocol: h2 or appProtocol: h2c or an existing endpoint is modified to change it to or from specifying one of these appProtocol values on a port. Dynamically configuring such an endpoint would require a way to set proto on the HAProxy server, and HAProxy's management interface does not provide any way to do so.

images/router/haproxy/conf/haproxy-config.template: Specify proto h2 for the server corresponding to an endpoint associated with a reencrypt route if the endpoint specifies the "h2" application protocol, and likewise for an endpoint associated with a non-TLS or edge-terminated route if the endpoint specifies the "h2c" application protocol.
pkg/router/template/configmanager/haproxy/backend.go (UpdateServerInfo): Add appProtocol parameter. If appProtocol is "h2" or "h2c", return an error.
pkg/router/template/configmanager/haproxy/manager.go (ReplaceRouteEndpoints): Return an error if an endpoint's AppProtocol is changed to or from "h2" or "h2c". Log AppProtocol and pass it to UpdateServerInfo.
pkg/router/template/plugin.go (createRouterEndpoints): Copy the AppProtocol field from the Kubernetes Endpoints object to the internal Endpoints object.
pkg/router/template/types.go (Endpoint): Add AppProtocol field.

* pkg/router/template/plugin_test.go (TestHandleTCPEndpoints): * pkg/router/template/router.go (createServiceUnitInternal): Fix typos in the godoc for these functions.

openshift-ci · 2021-08-26T19:43:57Z

@Miciah: This pull request references Bugzilla bug 1826225, which is invalid:

expected the bug to target the "4.9.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 1826225: Support edge-terminated h2 connections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Miciah · 2021-08-26T21:42:22Z

/bugzilla refresh

openshift-ci · 2021-08-26T21:42:36Z

@Miciah: This pull request references Bugzilla bug 1826225, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.9.0) matches configured target release for branch (4.9.0)
bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @quarterpin

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

frobware · 2021-08-31T17:53:47Z

@Miciah it would be worthwhile creating a PR that removes the restriction here https://github.com/openshift/origin/blob/master/test/extended/router/grpc-interop.go#L142 and verifying the behaviour.

openshift-merge-robot · 2021-09-06T07:20:53Z

/bugzilla refresh

The requirements for Bugzilla bugs have changed, recalculating validity.

openshift-ci · 2021-09-06T07:20:57Z

@openshift-merge-robot: This pull request references Bugzilla bug 1826225, which is invalid:

expected the bug to target the "4.10.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

The requirements for Bugzilla bugs have changed, recalculating validity.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Miciah · 2021-09-09T04:46:17Z

[sig-network-edge][Conformance][Area:Networking][Feature:Router] The HAProxy router should pass the gRPC interoperability tests passed using cluster-bot to test #328 and openshift/origin#26456 together: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1435803305925152768

Miciah · 2021-09-09T04:46:31Z

/bugzilla refresh

openshift-ci · 2021-09-09T04:48:09Z

@Miciah: An error was encountered querying GitHub for users with public email (aiyengar@redhat.com) for bug 1826225 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


Post "http://ghproxy/graphql": dial tcp 172.30.229.2:80: i/o timeout

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Miciah · 2021-09-09T04:56:22Z

I'm open to suggestions for alternatives to using "h2c" for appProtocol. I am using "h2c", even though it is nonstandard, because it unambiguously conveys both "HTTP/2" and "cleartext" and I could not find a standard name to use.

frobware · 2021-09-09T11:32:06Z

images/router/haproxy/conf/haproxy-config.template

@@ -620,7 +620,8 @@ backend {{ genBackendNamePrefix $cfg.TLSTermination }}:{{ $cfgIdx }}
              {{- range $idx, $endpoint := processEndpointsForAlias $cfg $serviceUnit (env "ROUTER_BACKEND_PROCESS_ENDPOINTS" "") }}
  server {{ $endpoint.ID }} {{ $endpoint.IP }}:{{ $endpoint.Port }} cookie {{ $endpoint.IdHash }} weight {{ $weight }}
                {{- if (eq $cfg.TLSTermination "reencrypt") }} ssl
-                  {{- if not (isTrue $router_disable_http2) }} alpn h2,http/1.1
+                  {{- if eq $endpoint.AppProtocol "h2c" }} proto h2


How does this work (or does this work) if HTTP/2 is disabled?

Adding proto h2 to the server stanza affects the protocol used for the connection from HAProxy to the backend and doesn't affect the protocol used for the connection from the client to the frontend, which is governed by the sslbindconf in the certificate map.

frobware · 2021-09-09T11:36:57Z

I'm open to suggestions for alternatives to using "h2c" for appProtocol. I am using "h2c", even though it is nonstandard, because it unambiguously conveys both "HTTP/2" and "cleartext" and I could not find a standard name to use.

A quick search gave me:

H2c is established protocol shorthand for HTTP/2 initiated by a
HTTP/1.1 Upgrade header sent over cleartext communication.

Miciah · 2021-09-11T20:26:47Z

I'm open to suggestions for alternatives to using "h2c" for appProtocol. I am using "h2c", even though it is nonstandard, because it unambiguously conveys both "HTTP/2" and "cleartext" and I could not find a standard name to use.

A quick search gave me:
H2c is established protocol shorthand for HTTP/2 initiated by a
HTTP/1.1 Upgrade header sent over cleartext communication.

It's "established" and codified for use in TLS and HTTP headers by RFC 7540. However, the API godoc for AppProtocol specifically mentions RFC 6335 and IANA, which do not have "h2c".

openshift-ci · 2021-09-13T21:29:42Z

@Miciah: This pull request references Bugzilla bug 1826225, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.10.0) matches configured target release for branch (4.10.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @quarterpin

In response to this:

Bug 1826225: Support edge-terminated h2 connections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

frobware · 2021-09-14T08:22:03Z

Latest push changes the logic to check for appProtocol: h2 for reencrypt routes because @alebedev87 pointed out that for reencrypt routes, we're using "h2" (HTTP/2 over TLS), not "h2c" (cleartext HTTP/2).

Can we extend the tests that we have to show that this would have failed and is now correct?

Check the appProtocol field on ports in Kubernetes Endpoints objects. If a non-TLS or edge-terminated route is associated with an endpoint with appProtocol: h2c, configure HAProxy to use HTTP/2 for the endpoint. Add checks in the dynamic config manager to force a full reload when a new endpoint specifies appProtocol: h2c or an existing endpoint is modified to change it to or from specifying appProtocol: h2c on a port. Dynamically configuring such an endpoint would require a way to set proto on the HAProxy server, and HAProxy's management interface does not provide any way to do so. This commit fixes bug 1826225. https://bugzilla.redhat.com/show_bug.cgi?id=1826225 * images/router/haproxy/conf/haproxy-config.template: Specify "proto h2" for the server corresponding to an endpoint associated with a non-TLS or edge-terminated route if the endpoint specifies the "h2c" application protocol. * pkg/router/template/configmanager/haproxy/backend.go (UpdateServerInfo): Add appProtocol parameter. If appProtocol is "h2c", return an error. * pkg/router/template/configmanager/haproxy/manager.go (ReplaceRouteEndpoints): Return an error if an endpoint's AppProtocol is changed to or from "h2c". Log AppProtocol and pass it to UpdateServerInfo. * pkg/router/template/plugin.go (createRouterEndpoints): Copy the AppProtocol field from the Kubernetes Endpoints object to the internal Endpoints object. * pkg/router/template/types.go (Endpoint): Add AppProtocol field.

Miciah · 2021-09-24T23:26:34Z

Latest push drops the logic for reencrypt routes, per recent discussions. I tested the latest version using cluster-bot, and job test e2e openshift/router#328,openshift/origin#26456 succeeded.

frobware · 2021-10-05T18:27:12Z

/lgtm

openshift-ci · 2021-10-05T18:27:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: frobware, Miciah

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Miciah,frobware]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-10-05T19:11:53Z

/retest-required