Changed headers handling

[TiARETiK]

TLDR: While using Crow v1.1 our team have encountered a problem, where using a big amount of heders in OPTION request response caused those to be parsed incorrectly by a number of user-end systems, including Curl. After spending quite some time on debugging, we still couldn't fully understand the reasons for such a behavior, yet we managed to produce a fix, which we humbly ask to accept to the main Crow branch.

So. We are using Crow for backend of a web app. We've migrated to version 1.1 and encountered an unexpected and quite problematic behavior, where GnomeWeb web browser mishandled login form. After some debugging, we established the following facts :

1. GnomeWeb and Curl couldn't correctly parse responses
2. Curl request:

curl 'http://localhost:8080/auth' -X 'OPTIONS' -H 'Accept: application/json, text/plain, */*' -H 'Content-Type: application/json' -H 'Origin: http://some.site' -H 'Referer: http://some.site/' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15' --data-binary '{"login":"admin","password":"1234"}' -v

Curl output

*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> OPTIONS /auth HTTP/1.1
> Host: localhost:8080
> Accept: application/json, text/plain, */*
> Content-Type: application/json
> Origin: http://some.site
> Referer: http://some.site/
> Sec-Fetch-Dest: empty
> Sec-Fetch-Mode: cors
> Sec-Fetch-Site: cross-site
> User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
> Content-Length: 35
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Origin: *
* Connection #0 to host localhost left intact

Expected Curl output

*   Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> OPTIONS /auth HTTP/1.1
> Host: localhost:8080
> Accept: application/json, text/plain, */*
> Content-Type: application/json
> Origin: http://ukazka.softcom.su
> Referer: http://ukazka.softcom.su/
> Sec-Fetch-Dest: empty
> Sec-Fetch-Mode: cors
> Sec-Fetch-Site: cross-site
> User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15
> Content-Length: 35
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Access-Control-Allow-Headers: *
< Access-Control-Allow-Methods: *
< Access-Control-Allow-Origin: *
< Access-Control-Expose-Headers: X-Total-Pages, X-Total-Entries
< Allow: OPTIONS, HEAD, POST
< Content-Length: 0
< Server: Crow/master
< Date: Fri, 19 Apr 2024 09:25:41 GMT
< 
* Connection #0 to host localhost left intact

3. Some apps (specifically, Mozilla Firefox web browser) still handled given responses correctly.

After some work we figured out, that not all of the headers were parsed. Also, exchange of headers positions in response resulted in the first ones being parsed correctly, while later ones were ignored. Thus we concluded, that problem was not connected to some specific header being sent incorrectly, but to the structure of response in general. We couldn't quite figure out the underlying reason for such behavior, yet the proposed change seams to fix our problem, and we believe that it can be useful to others for as long as the proper solution is not implemented.

[bugdea1er]

@gittiver could you look into this please? I have the same problem and this solves it. Although this doesn't seem like the right solution.

[luca-schlecker]

I agree with @bugdea1er that this doesn't seem like the correct solution.

I am quite short on time but I'd be willing to invest a little bit of it.

Could you please provide a simple example program producing this error so I can try and investigate?