HTTP/2 to support asynchronous SETTINGS ACK #9069

Scottmitch · 2019-04-17T16:41:02Z

Motivation:
The HTTP/2 codec will synchronously respond to a SETTINGS frame with a SETTINGS
ACK before the application sees the SETTINGS frame. The application may need to
adjust its state depending upon what is in the SETTINGS frame before applying
the remote settings and responding with an ACK (e.g. to adjust for max
concurrent streams). In order to accomplish this the HTTP/2 codec should allow
for the application to opt-in to sending the SETTINGS ACK.

Modifications:

DefaultHttp2ConnectionDecoder should support a mode where SETTINGS frames can
be queued instead of immediately applying and ACKing.
DefaultHttp2ConnectionEncoder should attempt to poll from the queue (if it
exists) to apply the earliest received but not yet ACKed SETTINGS frame.
AbstractHttp2ConnectionHandlerBuilder (and sub classes) should support a new
option to enable the application to opt-in to managing SETTINGS ACK.

Result:
HTTP/2 allows for asynchronous SETTINGS ACK managed by the application.

Motivation: The HTTP/2 codec will synchronously respond to a SETTINGS frame with a SETTINGS ACK before the application sees the SETTINGS frame. The application may need to adjust its state depending upon what is in the SETTINGS frame before applying the remote settings and responding with an ACK (e.g. to adjust for max concurrent streams). In order to accomplish this the HTTP/2 codec should allow for the application to opt-in to sending the SETTINGS ACK. Modifications: - DefaultHttp2ConnectionDecoder should support a mode where SETTINGS frames can be queued instead of immediately applying and ACKing. - DefaultHttp2ConnectionEncoder should attempt to poll from the queue (if it exists) to apply the earliest received but not yet ACKed SETTINGS frame. - AbstractHttp2ConnectionHandlerBuilder (and sub classes) should support a new option to enable the application to opt-in to managing SETTINGS ACK. Result: HTTP/2 allows for asynchronous SETTINGS ACK managed by the application.

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java

codec-http2/src/test/java/io/netty/handler/codec/http2/Http2MultiplexCodecTransportTest.java

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionEncoder.java

normanmaurer · 2019-04-18T04:41:15Z

codec-http2/src/test/java/io/netty/handler/codec/http2/Http2MultiplexCodecTransportTest.java

+                    @Override
+                    public void channelRead(ChannelHandlerContext ctx, Object msg) {
+                        if (msg instanceof Http2SettingsAckFrame) {
+                            serverAckLatch.countDown();


nit: maybe also verify we do not receive more of these frame as expected.

I changed the test to more carefully verify the first ACK and the second ACK, but I'll just leave it at that for now. Verifying more may require a timeout and this would be somewhat unreliable.

normanmaurer

LGTM... 2 nits

codec-http2/src/test/java/io/netty/handler/codec/http2/Http2MultiplexCodecTransportTest.java

...-http2/src/main/java/io/netty/handler/codec/http2/AbstractHttp2ConnectionHandlerBuilder.java

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2SettingsAckFrame.java

normanmaurer · 2019-04-18T15:16:19Z

still LGTM

Scottmitch · 2019-04-18T23:15:07Z

build failure attributed to #9075

Scottmitch · 2019-04-18T23:15:15Z

test this please

…ally

ejona86 · 2019-04-24T17:04:13Z

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java

+     *                    h2 semantics on top of the frames.
+     * @param requestVerifier Determines if push promised streams are valid.
+     * @param autoAckSettings {@code false} to disable automatically applying and sending settings acknowledge frame.
+     * In this case see see {@link #pollReceivedSettings()} which is expected to be consumed by the


Please indent at least 2 spaces.

You missed the next two lines.

ejona86 · 2019-04-24T17:11:47Z

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionEncoder.java


    public DefaultHttp2ConnectionEncoder(Http2Connection connection, Http2FrameWriter frameWriter) {
        this.connection = checkNotNull(connection, "connection");
        this.frameWriter = checkNotNull(frameWriter, "frameWriter");
+        remoteSettingsSupplier = EmptyHttp2SettingsReceivedSupplier.INSTANCE;


This scares me. It is really subtle. If a user sets autoAckSettingsFrame(false), then they must also set this, otherwise settings management will silently break and be horrible to debug. I see AbstractHttp2ConnectionHandlerBuilder configures this, but that's not the only way these can be connected.

There's quite a bit of plumbing here in general. It seems it would be better to have encoder.writeSettingsAck() be passed the Settings that is being Acked. That would mean we'd also remove remoteSettings(). Yes, that breaks API, but both methods can only be sanely called from the Decoder today and I doubt many people have implemented their own Decoder.

It would mean that Http2FrameCodec would need the Settings queue, but it would be an internal detail and not cross API boundaries.

if a user sets autoAckSettingsFrame(false), then they must also set this..

A user that provides their own encoder/decoer on the builder will not be able to use autoAckSettingsFrame on the builder (the builder will throw). The builder exposes autoAckSettingsFrame and this is how it is implemented under the hood. The remoteSettingsSupplier is an implementation detail from the perspective of autoAckSettingsFrame on the builder, and shouldn't lead to an inconsistent state.

It seems it would be better to have encoder.writeSettingsAck() be passed the Settings that is being Acked.

I don't think passing the settings to ACK into encoder.writeSettingsAck() is a good thing because it may lead to folks trying to ACK settings out of order, which would result in protocol violations. This would be more likely now that async settings ACK is permitted. The HTTP/2 protocol doesn't allow for ACKing specific settings frames. Each ACK applies to the earliest sent SETTINGs frame that has not yet been ACKed.

That would mean we'd also remove remoteSettings(). Yes, that breaks API.

I agree the existing dependencies between encoder/decoder and associated implementation is not ideal. However I don't want to change the existing design (which already has this dependency for applying local settings sent from the encoder, in the decoder when an ACK is received, see Http2ConnectionEncoder) with this PR. I don't think this warrants breaking the APIs for 4.1 as the approach is consistent (protocol logic lives in encoder/decoder), but we should revisit for 5.0 when we move everything to the frame/multiplex codec.

A user that provides their own encoder/decoer on the builder

Not every user uses the builder. It is not required. I agree that if you use the builder things look fine.

I don't think passing the settings to ACK into encoder.writeSettingsAck() is a good thing because it may lead to folks trying to ACK settings out of order

Shoot. Yeah. Then maybe can the queue be moved to the encoder? The decoder already references the encoder, so that seems much easier. I'm imagining code like:

public void onSettingsRead(final ChannelHandlerContext ctx, Http2Settings settings) throws Http2Exception { encoder.remoteSettingsReceived(settings); if (autoAckSettings) { encoder.writeSettingsAck(ctx, ctx.newPromise()); } }

This doesn't seem like an invasive re-design, any more than this PR is already doing.

Then maybe can the queue be moved to the encoder? The decoder already references the encoder, so that seems much easier

IIUC we are no longer breaking APIs/behavior for 4.1 so we cannot modify Http2ConnectionEncoder directly. Http2ConnectionEncoder has a remoteSettings method but queueing in the implementation of this method would break the semantics of this method. I will make some modifications to preserve the existing API/semantics and put the queue in the encoder. PTAL

codec-http2/src/main/java/io/netty/handler/codec/http2/Http2MultiplexCodecBuilder.java

ejona86

outstandingRemoteSettingsQueue puts the encoder in a different "mode", but any pain seems to be to maintain API compatibility and is relatively small and contained. Thank you!

Scottmitch · 2019-04-25T22:13:42Z

thanks for review @ejona86! agreed we should do this differently in 5.0 but lets accept the pain for 4.1.

normanmaurer · 2019-04-26T07:18:47Z

@Scottmitch can you open an issue for 5 so we do not forget ?

normanmaurer · 2019-04-28T09:39:56Z

I will just cherry-pick it into master as well for now to keep the code-base as similar as possible and we can revisit when we tackle http2 for Netty 5.

Motivation: The HTTP/2 codec will synchronously respond to a SETTINGS frame with a SETTINGS ACK before the application sees the SETTINGS frame. The application may need to adjust its state depending upon what is in the SETTINGS frame before applying the remote settings and responding with an ACK (e.g. to adjust for max concurrent streams). In order to accomplish this the HTTP/2 codec should allow for the application to opt-in to sending the SETTINGS ACK. Modifications: - DefaultHttp2ConnectionDecoder should support a mode where SETTINGS frames can be queued instead of immediately applying and ACKing. - DefaultHttp2ConnectionEncoder should attempt to poll from the queue (if it exists) to apply the earliest received but not yet ACKed SETTINGS frame. - AbstractHttp2ConnectionHandlerBuilder (and sub classes) should support a new option to enable the application to opt-in to managing SETTINGS ACK. Result: HTTP/2 allows for asynchronous SETTINGS ACK managed by the application.

normanmaurer · 2019-04-28T09:49:14Z

Ok done ... also cherry-picked ec62af0

Scottmitch requested a review from normanmaurer April 17, 2019 16:41

normanmaurer requested changes Apr 17, 2019

View reviewed changes

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java Outdated Show resolved Hide resolved

normanmaurer requested changes Apr 17, 2019

View reviewed changes

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java Outdated Show resolved Hide resolved

normanmaurer reviewed Apr 17, 2019

View reviewed changes

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java Outdated Show resolved Hide resolved

normanmaurer reviewed Apr 17, 2019

View reviewed changes

codec-http2/src/test/java/io/netty/handler/codec/http2/Http2MultiplexCodecTransportTest.java Outdated Show resolved Hide resolved

Scottmitch added 2 commits April 17, 2019 13:07

release read messages

fed80bd

review comments

55699c1

normanmaurer reviewed Apr 17, 2019

View reviewed changes

codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionEncoder.java Outdated Show resolved Hide resolved

normanmaurer requested a review from ejona86 April 17, 2019 20:21

review comments

45d06be

Scottmitch requested a review from normanmaurer April 17, 2019 21:24

normanmaurer reviewed Apr 18, 2019

View reviewed changes

normanmaurer approved these changes Apr 18, 2019

View reviewed changes

codec-http2/src/test/java/io/netty/handler/codec/http2/Http2MultiplexCodecTransportTest.java Show resolved Hide resolved

trustin reviewed Apr 18, 2019

View reviewed changes

review comments

eb0848d

introduce Http2SettingReceivedSupplier to avoid exposing Queue extern…

7c0b602

…ally

ejona86 reviewed Apr 24, 2019

View reviewed changes

Scottmitch added 2 commits April 24, 2019 23:06

review comments

0dbb64c

move queue to encoder

a0b0f6e

Scottmitch requested a review from ejona86 April 25, 2019 22:06

ejona86 approved these changes Apr 25, 2019

View reviewed changes

Scottmitch merged commit b3dba31 into netty:4.1 Apr 25, 2019

Scottmitch deleted the async_settings_ack branch April 25, 2019 22:52

normanmaurer added this to the 4.1.36.Final milestone Apr 28, 2019

normanmaurer assigned Scottmitch Apr 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HTTP/2 to support asynchronous SETTINGS ACK #9069

HTTP/2 to support asynchronous SETTINGS ACK #9069

Scottmitch commented Apr 17, 2019

normanmaurer Apr 18, 2019

Scottmitch Apr 18, 2019

normanmaurer left a comment

normanmaurer commented Apr 18, 2019

Scottmitch commented Apr 18, 2019

Scottmitch commented Apr 18, 2019

ejona86 Apr 24, 2019

Scottmitch Apr 25, 2019

ejona86 Apr 25, 2019

ejona86 Apr 24, 2019

Scottmitch Apr 25, 2019

ejona86 Apr 25, 2019

Scottmitch Apr 25, 2019

ejona86 left a comment

Scottmitch commented Apr 25, 2019

normanmaurer commented Apr 26, 2019

normanmaurer commented Apr 28, 2019

normanmaurer commented Apr 28, 2019

HTTP/2 to support asynchronous SETTINGS ACK #9069

HTTP/2 to support asynchronous SETTINGS ACK #9069

Conversation

Scottmitch commented Apr 17, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

normanmaurer left a comment

Choose a reason for hiding this comment

normanmaurer commented Apr 18, 2019

Scottmitch commented Apr 18, 2019

Scottmitch commented Apr 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejona86 left a comment

Choose a reason for hiding this comment

Scottmitch commented Apr 25, 2019

normanmaurer commented Apr 26, 2019

normanmaurer commented Apr 28, 2019

normanmaurer commented Apr 28, 2019