Add header support to HttpConnectProxiedSocketAddress

[lambdai]

Title
Can we add haproxy protocol at client side aside from server side proxy protocol (#8949) ?

Motivation

HTTP2 parser may not be trivial in a simple proxy. I experimented the codec of Envoy.
A TCP proxy reaches 8-10x throughput vs HTTP2 proxy because TCP proxy does not parse and materialize HTTP2 structures such as header. However, a reasonable TCP proxy wants more than TCP addresses to determine the route.

The HAProxy protocol can deliver resourceful attributes via TLV fields. It would be great if GRPC channel could support adding channel-wide attributes.

There is a Negotiator can be used to achieve "self-defined TCP payload ahead of H2 channel".

Alternative

Currently, only HTTP 1.1 CONNECT (aka https proxy) is offered in the GRPC channel, with a weak API.
E.g. only user and password are supported to define the HTTP1.1 CONNECT request, while HTTP1 supports any legit HTTP header names and values.

[ejona86]

HAProxy on client-side is very strange. I'm not aware of any client implementing it; it is intended for proxies. Isn't it much more straight-forward to extend HTTP CONNECT to allow other headers?

But that does lead to the question, "why do you need headers?" You don't actually say you need them, you just say the API is "weak." What functionality is it not able to provide? What is your use-case?

[lambdai]

Yeah, commonly grpc client specify the network name for resolvers, and service/method/attributes so that are materialized in the http2 stream headers.

1. Re "per connection headers"
   I am creating a reverse proxy for a specialized group of grpc requests,
   I want to avoid LB via http headers, because the header parsing is expensive for the volume of requests;
   The cardinality of the attributes is beyond the comfort zone of resolvers.

The proxy protocol or HTTP 1.1. CONNECT is used to deliver the attributes for all the GRPC requests transported in the TCP connection. I need them as the key to the load balance the served TCP connections, just as a common http2 proxy can dispatch each GRPC request according to the method.

2. Re weak HTTP proxy negotiator
   Correct me if I am wrong:
   The HTTP CONNECT established by the current GRPC java client can be summarized as

CONNECT <host> http1.1
user: <user>
password: <password>

What I demand is

CONNECT <host> http1.1
key1: value1
key2: value2
...

Whichever H1 CONNECT or HA proxy protocol is decided, I wish GRPC client can have some API/config so that the Negotiator could populate the key values provided by the client API.
Some non-critical factors that HAproxy protocol vs HTTP1 are

• HA proxy codec is easier, no response is needed.
• The TCP proxy I have chosen can retire HA proxy protocol codec after the negotiation, but cannot yet retire HTTP1.1 CONNECT.

[lambdai]

Regarding the headers, the candidates are caller, callee, and location such as host name and clusters.
Some are used by telemetry, and a small amount is used by routing.

The headers are not IP addresses that are only tenant supported by HA proxy protocol v1.

[ejona86]

I am creating a reverse proxy for a specialized group of grpc requests,

Does every request from a particular client use the same settings, or do they vary per-RPC and you need to route each RPC (on the client-side) to the correct tunnel?

The cardinality of the attributes is beyond the comfort zone of resolvers.

I don't understand what you mean here; I don't know what impact the attributes have to resolvers. Is the cardinality of the attributes or the attribute values which is high?

Regarding the headers, the candidates are caller, callee, and location such as host name and clusters. Some are used by telemetry, and a small amount is used by routing.

Caller might not be too ugly to put as the username, if you really had to. Callee, location, hostname, and clusters are generally encoded in the hostname passed to CONNECT. For example, you could use CONNECT host.cluster.service.example.com and separate the parts in the proxy. That's actually pretty normal in my mind (for the sort of folk that do things like this). I agree that the limitation there is adding new fields is hard. The things you mentioned though aren't all that strange though, so I don't know if it is all that likely more things would be needed in the future. Probably client's locality is biggest risk.

To extend gRPC to allow other headers, we'd add an immutable headers map to HttpConnectProxiedSocketAddress for a NameResolver to use. We'd then pass the headers to a different Netty constructor.

[lambdai]

I am creating a reverse proxy for a specialized group of grpc requests,

Does every request from a particular client use the same settings, or do they vary per-RPC and you need to route each RPC (on the client-side) to the correct tunnel?

The grpc client should group the rpc-calls so that all the rpc calls in the channel share the same setting.
I admit that's a burden at grpc user. I am happy to see if GRPC client impl would group the rpc-calls automatically, but I don't see an existing grpc component to achieve. Let me know if there is since I am newbie in the grpc.

The cardinality of the attributes is beyond the comfort zone of resolvers.

I don't understand what you mean here; I don't know what impact the attributes have to resolvers. Is the cardinality of the attributes or the attribute values which is high?

Sorry for the unclear. The number of all possible values combinations is high.
Say, I code encode "--" in the DNS name, so grpc client looks up the VIP name by the combination, and grpc proxy listens on the VIP.

Regarding the headers, the candidates are caller, callee, and location such as host name and clusters. Some are used by telemetry, and a small amount is used by routing.

Caller might not be too ugly to put as the username, if you really had to. Callee, location, hostname, and clusters are generally encoded in the hostname passed to CONNECT. For example, you could use CONNECT host.cluster.service.example.com and separate the parts in the proxy. That's actually pretty normal in my mind (for the sort of folk that do things like this). I agree that the limitation there is adding new fields is hard. The things you mentioned though aren't all that strange though, so I don't know if it is all that likely more things would be needed in the future. Probably client's locality is biggest risk.

Yeah, encoded these info into host name is what I am hacking.

To extend gRPC to allow other headers, we'd add an immutable headers map to HttpConnectProxiedSocketAddress for a NameResolver to use. We'd then pass the headers to a different Netty constructor.

Thank you! I also find this proxy detector is embedded in name resolution. It's likely I would extend the ProxiedAddress in the in-house resolver. Now that Netty HTTPHandler supports the arbitrary headers, it seems the eventual implementation is small.

[larry-safran]

@lambdai, are you offering to implement this feature or requesting that the gRPC team add it?

[lambdai]

@larry-safran I am happy to extend the ProxiedAddress, which is the HTTP 1 CONNECT solution.

GIven this solution is not aligned with the original HAProxy protocol, I'd like to create a new issue. WDYT?

[ejona86]

Adding a Map<String,String> to HttpConnectProxiedSocketAddress would be fine. I honestly think the hostname solution should work reasonably well and is the most common approach. But, sure, other headers could be added.

Netty (and potentially OkHttp) could be enhanced to read the header field from the HttpConnectProxiedSocketAddress.