Password containing newline fails when following http 302

[jonjohnsonjr]

This started happening in 7.62.0 (7.61.1 is fine). I narrowed it down to 46e1640 using git bisect, but I'm not familiar enough with curl to dig much deeper than that (it seems like a big change).

Using a password containing a newline works fine on the first request, but if the server returns a 302, something goes wrong when curl is reconstructing the right URL to follow, and fails with:

curl: (3) URL using bad/illegal format or missing URL

I reproduced this with a simple go server that listens on 8080 and redirects from /foo to /bar.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Location", "/bar")
		w.WriteHeader(http.StatusFound)
	})

	http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "bar\n")
	})

	log.Fatal(http.ListenAndServe(":8080", nil))
}

I have a ~/key.json file containing some "credentials" for a basic password:

{
  "hello": "world"
}

This fails because (I believe) key.json contains a newline:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:22:43 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

But using jq to drop newlines with --compact-output succeeds:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json | jq --compact-output .)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%22hello%22%3a%22world%22%7d@localhost:8080/bar'
* Found bundle for host localhost: 0x5d2f3589c0e0 [serially]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /bar HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 4
< Content-Type: text/plain; charset=utf-8
< 
bar
* Connection #0 to host localhost left intact

I did this

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
curl: (3) URL using bad/illegal format or missing URL

I expected the following

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
bar

curl/libcurl version

$ ./src/curl -V
curl 7.71.0-DEV (x86_64-pc-linux-gnu) libcurl/7.71.0-DEV OpenSSL/1.1.0l zlib/1.2.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

operating system

$ uname -a
Linux penguin 4.19.113-08528-g5803a1c7e9f9 #1 SMP PREEMPT Thu Apr 2 15:21:14 PDT 2020 x86_64 GNU/Linux

[sterchelen]

Hey,
First issue for me as trying to explain the why.
Look at lib/escape.c , we are authorizing anything except characters with an hexa code inferior to 0x20

curl/lib/escape.c

Lines 179 to 182 in fa4fbc5

	if(reject_ctrl && (in < 0x20)) {
	free(ns);
	return CURLE_URL_MALFORMAT;
	}

Here is the commit that introduces this behavior 75ca568
As @bagder 's commit mentioned, this is for security reason. Here is the secu advisory https://curl.haxx.se/docs/CVE-2012-0036.html

Hope that helps. I am sure that @bagder could add more context and information to my comment.

[bagder]

This looks like a duplicate of the issue we fixed with #5400 (600a8cd).

Can you try current master and see if the issue is fixed?

[jonjohnsonjr]

First issue for me as trying to explain the why.

This makes sense, thanks for the explanation.

Can you try current master and see if the issue is fixed?

Trying with current master still reproduces:

$ git rev-parse HEAD
fa4fbc533f6ce7e76fc005972a3100ef2a26cc6c

$ ./buildconf && ./configure --with-ssl && make

 $ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Fri, 19 Jun 2020 15:54:47 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

It looks like the fix you linked is being applied, per this line:

• Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'

FWIW, we managed to work around this by just constructing the Authorization header directly instead of using -u:

$ ./src/curl -L -H "Authorization: Basic $(echo _json_key:$(cat ~/key.json) | base64 -w0)" localhost:8080/foo
bar

Given we have a workaround, this isn't a big issue for us anymore, especially since the relevant code is a CVE mitigation. Feel free to close this issue if it's too much trouble.

[bagder]

This is an extention of a bug in libcurl's URL API: setting a perfectly valid URL like http://user%0aname:pass%0aword@www.example will return an error for curl_url_get() when trying to extract the user name and password - and have then URL decoded.

Update: I notice this is in fact even documented functionality...

[jonjohnsonjr]

I can confirm this is now fixed, thanks @bagder !

[bagder]

thanks for confirming @jonjohnsonjr !