7.83 breaks multi-line HTTP headers

[ppisar]

I noticed that curl 7.83 started to error on multi-line HTTP headers.

Sending a response like this:

HTTP/1.0 200 Ok
X-Response-message-code: authentication.error.userIsNotAuthenticated
X-Response-message-text:  Retry: Bad user name or password in second OTP
phase.
 This is very long header
 which should span to more lines.
   Surrounding LWS are meaning-less. 

to curl raises this error now:

$ curl http://localhost:2000/
curl: (8) Header without colon

This is very probably caused by #8610.

https://www.ietf.org/rfc/rfc2616.html#section-4.2 supports these headers:

Header fields can be
extended over multiple lines by preceding each extra line with at
least one SP or HT.

To my surprise https://www.ietf.org/rfc/rfc7230.html#section-3.2.4 deprecated them:

   Historically, HTTP header field values could be extended over
   multiple lines by preceding each extra line with at least one space
   or horizontal tab (obs-fold).  This specification deprecates such
   line folding except within the message/http media type
   ([Section 8.3.1](https://www.ietf.org/rfc/rfc7230.html#section-8.3.1)).
   A sender MUST NOT generate a message that includes
   line folding (i.e., that has any field-value that contains a match to
   the obs-fold rule) unless the message is intended for packaging
   within the message/http media type.
   [...]
   A user agent that receives an obs-fold in a response message that is
   not within a message/http container MUST replace each received
   obs-fold with one or more SP octets prior to interpreting the field
   value.

I found this regression with a test suite of libisds project. That project took the multi-line headers from a specification of a protocol it implements. Checking the latest specification of the protocol, the multi-line headers were replaced with single-line headers.

Still, would you consider restoring the old behavior? Or at least ignoring the consequent lines of the multi-lined headers as recommended with RFC 7230?

[bagder]

Yes, that was unintentional. I really had no thought about multi-line when I did that change. Also clearly we have no test case in curl for them.

I wouldn't mind bringing back some handling of them - even if the fact that this is noticed only just now indicates that this truly is a feature not widely used out there. After all, this header style has been deprecated for eight years already. (RFC 7230 is from 2014.)

If we bring back support for these, we should also add a test case to verify and I figure the header API needs to be either adjusted for it or clarified in docs how it handles them.

[ogayot]

Hello,
I just wanted to mention that the python-tornado is also affected by this change:
https://github.com/tornadoweb/tornado/blob/master/tornado/test/httpclient_test.py#L514

It came up as an autopkgtest regression in Ubuntu and Debian with the move to 7.83.

I'm opening a bug report in tornado so that the regression is addressed. Depending on the decision here (i.e., dropping the support for multi-line headers or reintroducing it), the fix will be different.

Thanks!
Olivier

[bagder]

So far, it seems this issue has only been found to break tests, not actual real world use cases.

[bagder]

If you build without headers API support, then maybe this little patch brings it back. I'll need to make a little more to do this "proper":

diff --git a/lib/http.c b/lib/http.c
index b215307dc..2c78c3b34 100644
--- a/lib/http.c
+++ b/lib/http.c
@@ -3797,15 +3797,20 @@ static CURLcode verify_header(struct Curl_easy *data)
     return CURLE_WEIRD_SERVER_REPLY;
   }
   if(k->headerline < 2)
     /* the first "header" is the status-line and it has no colon */
     return CURLE_OK;
-  ptr = memchr(header, ':', hlen);
-  if(!ptr) {
-    /* this is bad, bail out */
-    failf(data, "Header without colon");
-    return CURLE_WEIRD_SERVER_REPLY;
+  if((header[0] == ' ') || (header[0] == '\t'))
+    /* line folding */
+    ;
+  else {
+    ptr = memchr(header, ':', hlen);
+    if(!ptr) {
+      /* this is bad, bail out */
+      failf(data, "Header without colon");
+      return CURLE_WEIRD_SERVER_REPLY;
+    }
   }
   return CURLE_OK;
 }
 
 /*