Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

7.83 breaks multi-line HTTP headers #8844

Closed
ppisar opened this issue May 13, 2022 · 4 comments
Closed

7.83 breaks multi-line HTTP headers #8844

ppisar opened this issue May 13, 2022 · 4 comments

Comments

@ppisar
Copy link

@ppisar ppisar commented May 13, 2022

I noticed that curl 7.83 started to error on multi-line HTTP headers.

Sending a response like this:

HTTP/1.0 200 Ok
X-Response-message-code: authentication.error.userIsNotAuthenticated
X-Response-message-text:  Retry: Bad user name or password in second OTP
phase.
 This is very long header
 which should span to more lines.
   Surrounding LWS are meaning-less. 

to curl raises this error now:

$ curl http://localhost:2000/
curl: (8) Header without colon

This is very probably caused by #8610.

https://www.ietf.org/rfc/rfc2616.html#section-4.2 supports these headers:

Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT.

To my surprise https://www.ietf.org/rfc/rfc7230.html#section-3.2.4 deprecated them:

   Historically, HTTP header field values could be extended over
   multiple lines by preceding each extra line with at least one space
   or horizontal tab (obs-fold).  This specification deprecates such
   line folding except within the message/http media type
   ([Section 8.3.1](https://www.ietf.org/rfc/rfc7230.html#section-8.3.1)).
   A sender MUST NOT generate a message that includes
   line folding (i.e., that has any field-value that contains a match to
   the obs-fold rule) unless the message is intended for packaging
   within the message/http media type.
   [...]
   A user agent that receives an obs-fold in a response message that is
   not within a message/http container MUST replace each received
   obs-fold with one or more SP octets prior to interpreting the field
   value.

I found this regression with a test suite of libisds project. That project took the multi-line headers from a specification of a protocol it implements. Checking the latest specification of the protocol, the multi-line headers were replaced with single-line headers.

Still, would you consider restoring the old behavior? Or at least ignoring the consequent lines of the multi-lined headers as recommended with RFC 7230?

@bagder
Copy link
Member

@bagder bagder commented May 13, 2022

Yes, that was unintentional. I really had no thought about multi-line when I did that change. Also clearly we have no test case in curl for them.

I wouldn't mind bringing back some handling of them - even if the fact that this is noticed only just now indicates that this truly is a feature not widely used out there. After all, this header style has been deprecated for eight years already. (RFC 7230 is from 2014.)

If we bring back support for these, we should also add a test case to verify and I figure the header API needs to be either adjusted for it or clarified in docs how it handles them.

@ogayot
Copy link

@ogayot ogayot commented May 23, 2022

Hello,
I just wanted to mention that the python-tornado is also affected by this change:
https://github.com/tornadoweb/tornado/blob/master/tornado/test/httpclient_test.py#L514

It came up as an autopkgtest regression in Ubuntu and Debian with the move to 7.83.

I'm opening a bug report in tornado so that the regression is addressed. Depending on the decision here (i.e., dropping the support for multi-line headers or reintroducing it), the fix will be different.

Thanks!
Olivier

@bagder
Copy link
Member

@bagder bagder commented May 23, 2022

So far, it seems this issue has only been found to break tests, not actual real world use cases.

@bagder
Copy link
Member

@bagder bagder commented May 24, 2022

If you build without headers API support, then maybe this little patch brings it back. I'll need to make a little more to do this "proper":

diff --git a/lib/http.c b/lib/http.c
index b215307dc..2c78c3b34 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -3797,15 +3797,20 @@ static CURLcode verify_header(struct Curl_easy *data)
     return CURLE_WEIRD_SERVER_REPLY;
   }
   if(k->headerline < 2)
     /* the first "header" is the status-line and it has no colon */
     return CURLE_OK;
-  ptr = memchr(header, ':', hlen);
-  if(!ptr) {
-    /* this is bad, bail out */
-    failf(data, "Header without colon");
-    return CURLE_WEIRD_SERVER_REPLY;
+  if((header[0] == ' ') || (header[0] == '\t'))
+    /* line folding */
+    ;
+  else {
+    ptr = memchr(header, ':', hlen);
+    if(!ptr) {
+      /* this is bad, bail out */
+      failf(data, "Header without colon");
+      return CURLE_WEIRD_SERVER_REPLY;
+    }
   }
   return CURLE_OK;
 }
 
 /*

bagder added a commit that referenced this issue May 24, 2022
Folder header lines will now get passed through like before. The headers
API is adapted and will get and provide the content "un-folded".

Added test 1274 and extended test 1940 to verify.

Fixes #8844
Reported-by: Petr Pisar
bagder added a commit that referenced this issue May 24, 2022
Folded header lines will now get passed through like before. The headers
API is adapted and will get and provide the content "un-folded".

Added test 1274 and extended test 1940 to verify.

Fixes #8844
Reported-by: Petr Pisar
@bagder bagder closed this as completed in c9b60f0 May 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants