Add support for Transfer-Encoding: chunked

[vabresto]

I've spent the past few days trying to use Nim with OpenFaas (my first Nim project, just a proof-of-concept, see: https://gitlab.com/vabresto/nim-sudoku-faas/-/tree/master/) and had some issues with the Nim HTTP server because the OpenFaas of-watchdog proxy doesn't pass the Content-Length header, but rather uses chunked encoding (see: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding).

I've added support for this for my own uses and wanted to contribute it back to Nim.

I'd like to add tests but I'm not sure what the best way to do so is. Additionally, I can't actually compile and run the devel branch locally (OSX 15.6), so I made these changes based on the Nim 1.4 code I have locally.

Any feedback would be very welcome. Thanks!

[dom96]

Thank you for this!

I left some comments, please also include info about how you tested this locally (i.e. a test plan).

[vabresto]

I've been testing locally with my repo above, though the shortest possible test case I've used is:

# main.nim
import asynchttpserver, asyncdispatch

var server = newAsyncHttpServer()
proc cb(req: Request) {.async.} =
  await req.respond(Http200, "Got " & req.body)

waitFor server.serve(Port(7000), cb)

and then running
curl --data "hello=world" -H transfer-encoding:chunked http://127.0.0.1:7000/
which should output
Got hello=world instead of the current Content-Length required.

If there's a way to include this kind of curl testing in the nim tests, I'd be happy to do that too.

[dom96]

If there's a way to include this kind of curl testing in the nim tests, I'd be happy to do that too.

What might be the best way to test it is to capture the data that curl sends, then just create a simple Nim script that does the following:

• Creates an asynchttpserver instance
• sends the data that curl would send using a raw socket (you can use newAsyncSocket() + connect + send for this)

As I mentioned in the comment I think you need to test this more thoroughly manually as well. I will merge this anyway though, we can always make fixes to the broken implementation without any problems. The main thing for me is that the API either doesn't change or is changed in a way that makes sense :)

[dom96]

This looks suspicious to me, I have a feeling that chunked encoding doesn't delimit data with \r\n. You should test with larger files to make sure they work (if you're just testing with "hello world" then you're unlikely to run into this).

[vabresto]

@dom96 The two sources I've found with details on this topic say that the chunks are delimited with \r\n, see:

• (MDN): https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
• (Wikipedia): https://en.wikipedia.org/wiki/Chunked_transfer_encoding#Example

However, you were right to be suspicious! The code needs to discard these newlines, which I did not correctly do. I've fixed this (and added tests!) in this follow-up PR: #16678

I'd appreciate it if you could take a look and review that too! Thanks!

[dom96]

Also thank you for going the extra mile to resolve my comments, I saw that you had some trouble with the converter in IRC.

The reason it's there is for backwards compatibility: so that you don't need to change headers["Blah"] to headers["Blah"][0]. Indeed it shouldn't be removed.