Support both HTTP2 and HTTP1 for upstreams

[guydc]

Version

1.11.x (latest stable)

Is your feature request related to a problem? Please describe.

Gloo supports configuring either HTTP1 or HTTP2 options per upstream. If options for both protocols are provided, the Gloo translation fails.

Describe the solution you'd like

Envoy supports configuring both http_protocol_options and http2_protocol_options on the same cluster, as long as the non-default protocol_selection: USE_DOWNSTREAM_PROTOCOL option is used.

It should be possible to configure both HTTP1 and HTTP2 options for an upstream and define the upstream protocol selection policy that supports this configuration.

Gloo edge validating webhooks should not admit upstreams that configure both options, if appropriate protocol selection is not defined.

Describe alternatives you've considered

No response

Additional Context

No response

[ianmacclancy]

Most of this task I understand and am implementing

"Gloo edge validating webhooks should not admit upstreams that configure both options if appropriate protocol selection is not defined."

This line does not make sense to me. It is valid to pass both http1 and http2 on the same upstream and that is what this change is enabling. We currently allow users to pass both settings when useHttp2 is enabled

If this line of the description means you cannot pass both protocol selection enums, then that is doable

[guydc]

Hi @ianmacclancy. Currently, it's possible to specify both options for upstreams in Gloo Edge. However, this configuration is rejected by envoy (error message: cluster: Both HTTP1 and HTTP2 options may only be configured with non-default 'protocol_selection' values), leading to inconsistent behavior (quietly breaking the configuration from being applied, without any indication on the upstream or proxy status). Gloo Edge should protect from this sort of inconsistency.

Protection can be implemented with a webhook that explicitly forbids using both options without specifying an appropriate protocol selection policy. Alternatively, Gloo Edge can implicitly set protocol_selection: USE_DOWNSTREAM_PROTOCOL if both HTTP1 and HTTP2 options are specified on an upstream. However, this implicit behavior will not be future proof, as envoy also supports an ALPN-based upstream protocol selection in the newer Upstream HTTP Protocol Options Extension (Currently not used by Gloo). So, the policy should probably be provided by the user.

At the very least, the Gloo Edge docs should be clear about this behavior: if both options are required, protocol selection must also be configured.

[ianmacclancy]

For now, we won't make the decision to automatically recognize that multiple connection settings are passed and will reject the resource if both HTTP1 and HTTP2 are passed.

As part of the DOD for this task I am proposing a restructuring of the Upstream CRD to support the current envoy behavior and the deprecation of the cluster config protocol selection - ref: envoyproxy/envoy@7554d61

I agree that a better experience would be abstracting the protocol selection and working out end-user intentions based on what they pass rather than shifting the more complex logic to the end-user.

[ianmacclancy]

Final PR - this will update gloo ee with the changes - https://github.com/solo-io/solo-projects/pull/3788/files

[nfuden]

Closing per #6521 (comment)