Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Password containing newline fails when following http 302 #5582

Closed
jonjohnsonjr opened this issue Jun 18, 2020 · 6 comments
Closed

Password containing newline fails when following http 302 #5582

jonjohnsonjr opened this issue Jun 18, 2020 · 6 comments
Assignees

Comments

@jonjohnsonjr
Copy link

@jonjohnsonjr jonjohnsonjr commented Jun 18, 2020

This started happening in 7.62.0 (7.61.1 is fine). I narrowed it down to 46e1640 using git bisect, but I'm not familiar enough with curl to dig much deeper than that (it seems like a big change).

Using a password containing a newline works fine on the first request, but if the server returns a 302, something goes wrong when curl is reconstructing the right URL to follow, and fails with:

curl: (3) URL using bad/illegal format or missing URL

I reproduced this with a simple go server that listens on 8080 and redirects from /foo to /bar.

package main

import (
	"fmt"
	"log"
	"net/http"
)

func main() {
	http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Location", "/bar")
		w.WriteHeader(http.StatusFound)
	})

	http.HandleFunc("/bar", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintf(w, "bar\n")
	})

	log.Fatal(http.ListenAndServe(":8080", nil))
}

I have a ~/key.json file containing some "credentials" for a basic password:

{
  "hello": "world"
}

This fails because (I believe) key.json contains a newline:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:22:43 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

But using jq to drop newlines with --compact-output succeeds:

$ ./src/curl -v -L -u _json_key:"$(cat ~/key.json | jq --compact-output .)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%22hello%22%3a%22world%22%7d@localhost:8080/bar'
* Found bundle for host localhost: 0x5d2f3589c0e0 [serially]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /bar HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsiaGVsbG8iOiJ3b3JsZCJ9
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Thu, 18 Jun 2020 19:27:11 GMT
< Content-Length: 4
< Content-Type: text/plain; charset=utf-8
< 
bar
* Connection #0 to host localhost left intact

I did this

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
curl: (3) URL using bad/illegal format or missing URL

I expected the following

$ ./src/curl -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
bar

curl/libcurl version

$ ./src/curl -V
curl 7.71.0-DEV (x86_64-pc-linux-gnu) libcurl/7.71.0-DEV OpenSSL/1.1.0l zlib/1.2.8
Release-Date: [unreleased]
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

operating system

$ uname -a
Linux penguin 4.19.113-08528-g5803a1c7e9f9 #1 SMP PREEMPT Thu Apr 2 15:21:14 PDT 2020 x86_64 GNU/Linux
@sterchelen
Copy link
Contributor

@sterchelen sterchelen commented Jun 18, 2020

Hey,
First issue for me as trying to explain the why.
Look at lib/escape.c , we are authorizing anything except characters with an hexa code inferior to 0x20

curl/lib/escape.c

Lines 179 to 182 in fa4fbc5

if(reject_ctrl && (in < 0x20)) {
free(ns);
return CURLE_URL_MALFORMAT;
}

Here is the commit that introduces this behavior 75ca568
As @bagder 's commit mentioned, this is for security reason. Here is the secu advisory https://curl.haxx.se/docs/CVE-2012-0036.html

Hope that helps. I am sure that @bagder could add more context and information to my comment.

@bagder
Copy link
Member

@bagder bagder commented Jun 18, 2020

This looks like a duplicate of the issue we fixed with #5400 (600a8cd).

Can you try current master and see if the issue is fixed?

@jonjohnsonjr
Copy link
Author

@jonjohnsonjr jonjohnsonjr commented Jun 19, 2020

First issue for me as trying to explain the why.

This makes sense, thanks for the explanation.

Can you try current master and see if the issue is fixed?

Trying with current master still reproduces:

$ git rev-parse HEAD
fa4fbc533f6ce7e76fc005972a3100ef2a26cc6c

$ ./buildconf && ./configure --with-ssl && make

 $ ./src/curl -v -L -u _json_key:"$(cat ~/key.json)" localhost:8080/foo
*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
* Server auth using Basic with user '_json_key'
> GET /foo HTTP/1.1
> Host: localhost:8080
> Authorization: Basic X2pzb25fa2V5OnsKICAiaGVsbG8iOiAid29ybGQiCn0=
> User-Agent: curl/7.71.0-DEV
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Location: /bar
< Date: Fri, 19 Jun 2020 15:54:47 GMT
< Content-Length: 0
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'
* Closing connection -1
curl: (3) URL using bad/illegal format or missing URL

It looks like the fix you linked is being applied, per this line:

  • Issue another request to this URL: 'http://_json_key:%7b%0a%20%20%22hello%22%3a%20%22world%22%0a%7d@localhost:8080/bar'

FWIW, we managed to work around this by just constructing the Authorization header directly instead of using -u:

$ ./src/curl -L -H "Authorization: Basic $(echo _json_key:$(cat ~/key.json) | base64 -w0)" localhost:8080/foo
bar

Given we have a workaround, this isn't a big issue for us anymore, especially since the relevant code is a CVE mitigation. Feel free to close this issue if it's too much trouble.

@bagder bagder self-assigned this Jun 19, 2020
bagder added a commit that referenced this issue Jun 19, 2020
(except for the path part)

Reported-by: Jon Johnson Jr
Fixes #5582
@bagder bagder linked a pull request that will close this issue Jun 19, 2020
@bagder
Copy link
Member

@bagder bagder commented Jun 23, 2020

This is an extention of a bug in libcurl's URL API: setting a perfectly valid URL like http://user%0aname:pass%0aword@www.example will return an error for curl_url_get() when trying to extract the user name and password - and have then URL decoded.

Update: I notice this is in fact even documented functionality...

bagder added a commit that referenced this issue Jun 23, 2020
Reported-by: Jon Johnson Jr
Fixes #5582
bagder added a commit that referenced this issue Jun 24, 2020
Reported-by: Jon Johnson Jr
Fixes #5582
Closes #5592
@bagder bagder closed this in d5ed571 Jun 25, 2020
@jonjohnsonjr
Copy link
Author

@jonjohnsonjr jonjohnsonjr commented Jun 25, 2020

I can confirm this is now fixed, thanks @bagder !

@bagder
Copy link
Member

@bagder bagder commented Jun 25, 2020

thanks for confirming @jonjohnsonjr !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.