- Author(s): ctiller
- Approver: ejona
- Status: Draft
- Implemented in: n/a
- Last updated: 3/29/17
- Discussion at: https://groups.google.com/forum/#!topic/grpc-io/Ww8waYz_Nes
Add a HTTP2 extension to allow binary metadata (those suffixed with -bin) to be sent without the base64 encode/decode step.
gRPC allows binary metadata to be sent by applications. These metadata elements are keyed with a -bin suffix. When transmitted on the wire, since HTTP2 does not allow binary headers, we base64 encode these elements and Huffman compress them. On receipt, we transparently reverse the transformation.
This transformation is costly in terms of CPU, and there exist use cases where this transformation can become the CPU bottleneck for gRPC.
n/a
Expose a custom setting in our HTTP2 settings exchange: GRPC_ALLOW_TRUE_BINARY_METADATA = 0xfe03.
This setting is randomly chosen (to avoid conflicts with other extensions), and within the experimental range of HTTP extensions (see https://tools.ietf.org/html/rfc7540#section-11.3).
The setting can have the values 0 (default) or 1. If the setting is 1, then peers MAY use a 'true binary' encoding (described below), instead of the current base64 encoding for -bin metadata.
Implementations SHOULD transmit this setting only once, and as part of the first settings frame.
When transmitting metadata on a connection where the peer has specified GRPC_ALLOW_TRUE_BINARY_METADATA, instead of encoding using base64, an implementation MAY instead prefix a NUL byte to the metadata and transmit the metadata value in binary form.
Since this is a HTTP2 extension and other extensions might alias this extension id, it's possible that this becomes misconfigured. In that case, peers are required to RST_STREAM with http error PROTOCOL_ERROR (as required by https://tools.ietf.org/html/rfc7540#section-10.3). If a binary encoding was attempted and such a RST_STREAM is received without any other headers, implementations SHOULD retry the request with base64 encoding, and disable binary encoding for future requests. Verbosely logging this condition is encouraged.
Suppose we wanted to send metadata element 'foo-bin: 0x01' (ie a single byte containing '1').
Under base64, we'd send a http header 'foo-bin: AQ'. Under binary, we'd send 'foo-bin: 0x00 0x01' (ie prefixing a NUL byte and then sending the binary metadata value.)
Binary metadata transmission performance is critical for a number of applications.
Various workarounds were considered:
- Switching to base16 - this would require a backwards incompatible protocol change for gRPC, and has the disadvantage of bloating wire size, which additionally interacts badly with hpack.
- Adding a new suffix (-raw): this leaks further implementation details to application developers, and likely would still need a base64 workaround
A trial implementation in gRPC C core is underway and expected to be ready in coming weeks.
n/a