Add better guards again re-entrance in ByteToMessageDecoder #370

normanmaurer · 2018-04-28T11:49:04Z

Motivation:

When the ByteToMessageDecoder re-entranced in its decode(...) / decodeLast(...) methods it could be possible that the ordering of processing and so the seen bytes are mixed.

Modifications:

Refresh the buffer on each loop iteration and update the cumulationBuffers indicies.

Result:

More robust code.

normanmaurer · 2018-04-28T11:53:03Z

Sources/NIOHTTP1/HTTPDecoder.swift

        assert(HTTPMessageT.self == HTTPClientResponsePart.self || HTTPMessageT.self == HTTPServerRequestPart.self)
    }

+    deinit {


I used a deinit { } for it as decoderRemoved may be called while still in the decode(...) and so I can not modify the parser.

Lukasa · 2018-04-28T12:08:28Z

Sources/NIO/Codec.swift

+    /// Decode in a loop until there is nothing more to decode.
+    private func decodeLoop(ctx: ChannelHandlerContext, decodeFunc: (ChannelHandlerContext, inout ByteBuffer) throws -> DecodingState) {
+        assert(self.cumulationBuffer != nil)
+        ctx.withThrowingToFireErrorAndClose {


Probably better to hoist this up to the caller.

You mean let the error bubble up ?

Let decodeLoop throw, place the ctx.thenThrowingToFireErrorAndClose in the function that calls decodeLoop.

Lukasa · 2018-04-28T12:09:50Z

Sources/NIO/Codec.swift

+                let writerIndex = buffer.writerIndex
+                let result = try decodeFunc(ctx, &buffer)
+                if self.cumulationBuffer != nil {
+                    self.cumulationBuffer!.moveReaderIndex(forwardBy: readable - buffer.readableBytes)


Is this right? We’re passing a local copy of the buffer, but then mutating the cumulation buffer. Why not just preserve the buffer returned from the call to decode, especially as we’re bothering to pass it inout.

I think it’s right but I am all ears for better ideas. Can you show me some code with your suggestion as I don’t understand what exactly you suggest here

Basically I'm just saying why not just do self.cumulationBuffer = buffer?

Ah, I know why: the cumulationBuffer may have been mutated. That's awkward.

Hrm, I'm inclined to say that we should probably pass &buffer as a slice instead, and then come up with some custom logic to reconcile the changes from that slice with the parent.

Yeah it’s because of the reason you stated... a slice sounds ok, that said I am not sure if it really buys us anything

Lukasa · 2018-04-28T12:10:37Z

Sources/NIO/Codec.swift

-                buffer.discardReadBytes()
+        if self.cumulationBuffer != nil {
+            if self.cumulationBuffer!.readableBytes > 0 {
+                if self.shouldReclaimBytes(buffer: self.cumulationBuffer!) && self.cumulationBuffer!.discardReadBytes() && self.cumulationBuffer!.readableBytes == 0 {


Let’s make this clearer.

Ok let me break up the if statement and add some comments

normanmaurer · 2018-04-28T12:22:33Z

@swift-nio-bot test this please

normanmaurer · 2018-04-28T12:33:59Z

Also tests pass in docker locally... we may need to update timeouts

weissi · 2018-04-30T13:45:16Z

Sources/NIOHTTP1/HTTPDecoder.swift

+        parser.data = UnsafeMutableRawPointer(bitPattern: 0x0000deadbeef0000)
+
+        // Set the callbacks to nil as we dont need these anymore
+        settings.on_body = nil


this should have the same effect:

settings = http_parser_settings()

tomerd · 2018-04-30T20:39:42Z

@swift-nio-bot test this please

normanmaurer · 2018-05-04T17:47:58Z

Let me update the PR to only include the ByteToMessageDecoder changes.

Lukasa

Can we have a test for this change?

normanmaurer · 2018-05-04T18:00:53Z

@weissi @Lukasa PTAL again... Will add some more tests as well.

normanmaurer · 2018-05-04T18:01:31Z

@Lukasa haha you were faster then me (and I did not refresh yet). Yep will add a few tests.

weissi · 2018-05-04T19:27:23Z

@normanmaurer awesome!

18:30:29 info: 1000_reqs_1_conn: total number of mallocs: 46730
18:30:29 info: 1_reqs_1000_conn: allocations not freed: 0
18:30:29 info: 1_reqs_1000_conn: total number of mallocs: 775997
18:30:29 info: ping_pong_1000_reqs_1_conn: allocations not freed: 0
18:30:29 info: ping_pong_1000_reqs_1_conn: total number of mallocs: 4513
18:30:29 info: bytebuffer_lots_of_rw: allocations not freed: 0
18:30:29 info: bytebuffer_lots_of_rw: total number of mallocs: 3011
18:30:29 info: future_lots_of_callbacks: allocations not freed: 0
18:30:29 info: future_lots_of_callbacks: total number of mallocs: 99001

Please edit the docker compose files and lower the limits once again. Just add the changes to this PR

Lukasa · 2018-05-05T11:37:57Z

Sources/NIO/Codec.swift

-            if self.shouldReclaimBytes(buffer: buffer) {
-                buffer.discardReadBytes()
+        // Discard the cumulationBuffer or discard read bytes if needed.
+        guard self.cumulationBuffer != nil else {


Any reason not to use guard let here?

Because we want to act on the self.cumulationBuffer on the following lines and not on a different copy ?

Lukasa · 2018-05-05T11:38:54Z

Sources/NIO/Codec.swift

+        }
+
+        if self.shouldReclaimBytes(buffer: self.cumulationBuffer!) && self.cumulationBuffer!.discardReadBytes() {
+            if self.cumulationBuffer!.readableBytes == 0 {


Any reason not to just hoist this check up? It won’t be changed by reclaiming bytes.

@Lukasa you are right this can be removed.

normanmaurer · 2018-05-07T07:48:19Z

Sources/NIO/Codec.swift


+    /// Decode in a loop until there is nothing more to decode.
+    private func decodeLoop(ctx: ChannelHandlerContext, decodeFunc: (ChannelHandlerContext, inout ByteBuffer) throws -> DecodingState) throws {
+        while var slice = self.cumulationBuffer?.slice(), slice.readableBytes > 0 {


@weissi @Lukasa so I started to write some unit tests and notice we can not do this.. The problem here is that it is possible that decodeLast will be triggered from within decodeFunc (due a close(...) call for example). In this case decodeLast will by default call decode(...) which will then see the same bytes again even if decode(...) before increased the readerIndex as we had no chance to replicate these changes to the cumulationBuffer yet :(

Yeah, there's no way to entirely prevent re-entrancy without removing decodeLast.

yeah... so I wonder what we should do here in the case... any suggestions ?

I mean, I'm inclined to sit on it and wait until 2.0, when decodeLast is removed. In the meantime we can recommend users override decodeLast or channelInactive.

@Lukasa when you say sit on it you suggest just not pass in a slice but self.cumulationBuffer or pass in the slice (as I did here) ?

I mean, it doesn't matter what you do, the issue is still the same: decodeLast is not safe from avoid reentrancy. So you may as well keep using the slice.

Lukasa · 2018-05-08T13:43:42Z

Sources/NIO/Codec.swift

+            }
+
+            guard slice.writerIndex == sliceWriterIndex else {
+                fatalError("Writing to the buffer is not allowed")


Minor nit, but I tend to prefer these to be preconditionFailure.

sure why not... I dont care and I think at the end it makes no difference as long as it crashes :)

Lukasa · 2018-05-08T13:45:36Z

Tests/NIOTests/CodecTest.swift

+            typealias InboundOut = ByteBuffer
+
+            var cumulationBuffer: ByteBuffer?
+            var triggeredReentrace: Bool = false


nit: "reentrance"

Lukasa · 2018-05-08T13:47:39Z

Tests/NIOTests/CodecTest.swift

+                }
+
+                if !self.triggeredReentrace {
+                    self.triggeredReentrace = true


Rather than set this here, it might be better to let the re-entrant call set it (where you currently have an XCTAssertTrue) and then check it at the end of the test. Otherwise this test could pass if you never re-entered at all.

Lukasa

Cool, I'm basically happy. @weissi?

weissi · 2018-05-08T14:40:59Z

Sources/NIO/Codec.swift

+/// 1. Moving the reader index forward persists across calls. When your method returns, if the reader index has advanced, those bytes are considered "consumed" and will not be available in future calls to `decode`.
+///    Please note, however, that the numerical value of the `readerIndex` itself is not preserved, and may not be the same from one call to the next. Please do not rely on this numerical value: if you need
+///    to recall where a byte is relative to the `readerIndex`, use an offset rather than an absolute value.
+/// 2. Mutating the bytes in the buffer will cause a `fatalError` and so is not allowed. You are only allowed to move the readerIndex forward or consume from the buffer.


should we say 'mutating the bytes or the readerIndex will cause undefined behaviour and likely crash your program' or something?

weissi

mostly nits and can you lower the allocation limits?

weissi · 2018-05-08T14:42:53Z

Sources/NIO/Codec.swift

+                break
+            }
+
+            guard slice.writerIndex == sliceWriterIndex else {


why not just precondition(slice.writeIndex == sliceWriterIndex, "...")?

weissi

nice one, ta

Motivation: When the ByteToMessageDecoder re-entranced in its decode(...) / decodeLast(...) methods it could be possible that the ordering of processing and so the seen bytes are mixed. Modifications: - Refresh the buffer (and take a slice) on each loop iteration and update the cumulationBuffers indicies. Result: More robust code.

normanmaurer requested a review from weissi April 28, 2018 11:51

normanmaurer commented Apr 28, 2018

View reviewed changes

Lukasa requested changes Apr 28, 2018

View reviewed changes

weissi reviewed Apr 30, 2018

View reviewed changes

normanmaurer force-pushed the byte_to_message_decoder branch from 5a9c7f6 to 5af4c27 Compare May 4, 2018 17:27

normanmaurer changed the title ~~[DONT-MERGE_YET] ByteToMessageDecoder improvements / HTTPDecoder allocations~~ Add better guards again re-entrance in ByteToMessageDecoder May 4, 2018

normanmaurer force-pushed the byte_to_message_decoder branch from 5af4c27 to a7d4ff3 Compare May 4, 2018 17:49

Lukasa requested changes May 4, 2018

View reviewed changes

normanmaurer force-pushed the byte_to_message_decoder branch 3 times, most recently from 4a0a64a to 58ff97f Compare May 4, 2018 18:00

normanmaurer force-pushed the byte_to_message_decoder branch from 58ff97f to d1f08f5 Compare May 4, 2018 18:01

Lukasa requested changes May 5, 2018

View reviewed changes

normanmaurer commented May 7, 2018

View reviewed changes

normanmaurer force-pushed the byte_to_message_decoder branch from d01f35c to 8abacf9 Compare May 8, 2018 13:27

Lukasa requested changes May 8, 2018

View reviewed changes

Lukasa approved these changes May 8, 2018

View reviewed changes

weissi reviewed May 8, 2018

View reviewed changes

weissi requested changes May 8, 2018

View reviewed changes

weissi approved these changes May 8, 2018

View reviewed changes

normanmaurer force-pushed the byte_to_message_decoder branch from 832de2f to 35783b7 Compare May 8, 2018 16:17

normanmaurer self-assigned this May 8, 2018

normanmaurer added this to the 1.7.0 milestone May 8, 2018

normanmaurer merged commit 4cbafde into apple:master May 8, 2018

normanmaurer deleted the byte_to_message_decoder branch May 8, 2018 16:29

Lukasa added the 🔨 semver/patch No public API change. label May 11, 2018

Add better guards again re-entrance in ByteToMessageDecoder #370

Add better guards again re-entrance in ByteToMessageDecoder #370

Uh oh!

Conversation

normanmaurer commented Apr 28, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

normanmaurer commented Apr 28, 2018

Uh oh!

normanmaurer commented Apr 28, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomerd commented Apr 30, 2018

Uh oh!

normanmaurer commented May 4, 2018

Uh oh!

Lukasa left a comment

Choose a reason for hiding this comment

Uh oh!

normanmaurer commented May 4, 2018

Uh oh!

normanmaurer commented May 4, 2018

Uh oh!

weissi commented May 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lukasa left a comment

normanmaurer commented Apr 28, 2018 •

edited

Loading

weissi commented May 4, 2018 •

edited

Loading