Consider migrating from net/http to valyala/fasthttp

[mardukbp]

I am using k6 for load-testing an application developed by a client. Using k6/http, which is based on net/http, I met some problems and I ended up developing my own http module based on valyala/fasthttp. Here is why:

• net/http normalizes header names (using title case). In fasthttp this can be turned off with req.Header.DisableNormalizing()

This is relevant for headers named MY_CUSTOM_HEADER.

• In net/http headers are represented as a dictionary (map) of lists. fasthttp supports that and also a dictionary of strings. This is relevant for my use case.

• In fasthttp Request has the methods SetBody (for binary content) and SetBodyString. I needed the former. I am aware that k6 currently has poor support for sending a binary body.

And finally, fasthttp claims to be much faster. Pretty good for a load-testing tool!

[na--]

We've investigated it before, but the biggest obstacle to adoption is that fasthttp still doesn't support HTTP/2... From it's README:

• Why fasthttp doesn't support HTTP/2.0 and WebSockets?

HTTP/2.0 support is in progress. WebSockets has been done already.

And looking at the repo, there doesn't seem to have been a lot of recent progress, so I'm not holding my breath for it being possible to switch any time soon... 😞

[mardukbp]

I see. Nevertheless I think that the issues I encountered with net/http should be addressed. While normalization of headers can be avoided by asigning a header directly, e.g. Header["MY_CUSTOM_HEADER"] = []string{"some_value"}, having a list on the right side is not acceptable for some servers. I think that net/http has a highly opinionated API, whereas fasthttp offers more flexibility. k6 could adopt this approach for its http module.

[na--]

IIRC, HTTP header names are supposed to be case-insensitive... So, yes, net/http is highly opinionated, but it can also be said that it follows the RFC, whereas server implementation that care about the case of header names probably aren't...

There are multiple other issues with the current k6/http API, so we plan to make a new JS API for HTTP in k6 eventually, to address all of them. I'm not sure when that will be, but when we create the overarching issue for discussing that new API, I'll also include this issue, though I can't promise that all possible cases can be addressed.

[mardukbp]

I can happily report that I managed to get my load tests running with stock k6. My custom k6 plugin implements functions that return []byte, which on the JS side is an object that when passed to k6/http.post is handled correctly.

The issue with the headers turns out to be localized to the cookies. I was using

var params = { cookies: { cookie_name : cookie_value } }

and I always got a 401 response. Then I realized that according to the documentation cookie_name becomes "cookie_name". So I entered the cookie name manually and the problem was solved. My bad. I was not familiar with JS dictionaries.

In conclusion, this issue can be closed.

[oleiade]

ref #1246