Windows specific failure with basic auth using userpwd curl option but not header #224

jefferis · 2020-05-14T00:06:28Z

I use curl via httr to access a web application used in brain mapping (https://github.com/natverse/rcatmaid). A colleague recently reported that he could not access his server which is protected by http basic authentication.

The gist of the problem is in this error message:

 h=curl::new_handle()
> h=handle_setopt(h, .list=list(userpwd = "catmaid:#DZaRJYrixKE*gFY", verbose=T))
> h=handle_setheaders(h, .list = list('X-Authorization' = 
+                                       'Token ee6e7fdb03a0d35b3a6f499d0f8f610686551d51'))
> res=curl_fetch_memory("https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations", h)

* Server auth using Basic with user 'catmaid'
> GET /catmaid/2/annotations HTTP/1Error in curl_fetch_memory("https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations",  : 
  Could not resolve host: catmaid

i.e. the basic auth username (catmaid) somehow gets mangled into the url and misinterpreted as the server URL.

After digging, I can report the following.

I only see the problem on windows not macosx
I have traced the problem as low as curl but I don't know if it is in curl or libcurl
If I base64 encode the user:password and add as a curl header like so:

h2=curl::new_handle()
h2=handle_setopt(h2, verbose=TRUE)
h2=handle_setheaders(h2, .list = list(
  'X-Authorization' = 'Token ee6e7fdb03a0d35b3a6f499d0f8f610686551d51',
  Authorization = paste("Basic", jsonlite::base64_enc("catmaid:#DZaRJYrixKE*gFY"))))
res=curl_fetch_memory("https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations", h2)
rawToChar(res$content)

then everything works just fine on windows and mac.

Not all paths on the server cause trouble.
updating to the github version of curl has no effect

I am really stumped. Can you give any advice? Note that the tokens / password info above are fake as I'm afraid I cannot provide the real credentials here, but the password in user:password really does have both a # and a * in it.

> sessionInfo()
R version 3.6.2 (2019-12-12)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=English_United Kingdom.1252  LC_CTYPE=English_United Kingdom.1252   
[3] LC_MONETARY=English_United Kingdom.1252 LC_NUMERIC=C                           
[5] LC_TIME=English_United Kingdom.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] curl_4.3

loaded via a namespace (and not attached):
[1] compiler_3.6.2 tools_3.6.2

The text was updated successfully, but these errors were encountered:

jeroen · 2020-05-14T08:49:25Z

There may have been a bug in the URL parser of the libcurl version that we shipped on Windows.

As a workaround, instead of setting userpwd can you set username and password as separate arguments:

h=handle_setopt(h, username="catmaid", password = "#DZaRJYrixKE*gFY")

jefferis · 2020-05-14T08:57:58Z

Thank you very much for the response Jeroen. How would you suggest do this from httr? I assume I can’t use the authenticate form but have to roll my own. Best, Greg.

…

Sent from my iPhone

On 14 May 2020, at 09:49, Jeroen Ooms ***@***.***> wrote: There may have been a bug in the URL parser of the libcurl version that we shipped on Windows. As a workaround, instead of setting userpwd can you set username and password as separate arguments: https://curl.haxx.se/libcurl/c/CURLOPT_USERNAME.html https://curl.haxx.se/libcurl/c/CURLOPT_PASSWORD.html h=handle_setopt(h, username="catmaid", password = "#DZaRJYrixKE*gFY") — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jeroen · 2020-05-14T08:59:32Z

Ah I didn't know httr uses userpwd. If you confirm this fixes the problem, I'll change it in httr.

jefferis · 2020-05-14T09:28:18Z

Thanks, Jeroen. I tried and unfortunately I still get the same error. Perhaps the username and password options eventually get combined into userpwd. So until now, the only workaround I could find was to add this header (as noted above, jsonlite to the rescue ;-)):

 Authorization = paste("Basic", jsonlite::base64_enc("catmaid:#DZaRJYrixKE*gFY"))

I guess I can do that like so in httr, but it seems unfortunate to have to by pass the normal authenticate mechanism.

library(httr)
fullconfig2 = add_headers(
  .headers = c(
    'X-Authorization' = 'Token ee6e7fdb03a0d35b3a6f499d0f8f610686551d51',
    Authorization = paste("Basic", jsonlite::base64_enc("catmaid:#DZaRJYrixKE*gFY"))
  )
)

 GET(
    "https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations",
    config = fullconfig2
  )

jefferis · 2020-05-14T09:28:58Z

PS a bit more info from verbose about how the URL is getting munged when things fail.

< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Thu, 14 May 2020 09:20:26 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Location: /catmaid/2/annotations/
< X-Robots-Tag: noindex, nofollow
< 
* Connection #0 to host fuzo-sthi2.biol.lu.se left intact
* Issue another request to this URL: 'https://catmaid/#DZaRJYrixKE*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations/'
* NTLM-proxy picked AND auth done set, clear picked!
* Could not resolve host: catmaid

jeroen · 2020-05-14T09:49:59Z

@bagder do you recall a bug in libcurl ~ 7.64.1 that would cause the userpwd to be parsed incorrectly?

bagder · 2020-05-14T11:10:43Z

No, I cannot recall having seen any bug like this. Is it possible to see the wrong header emitted as compared to how the working header looks like? Maybe that will help...

jefferis · 2020-05-14T11:49:21Z

Hi @bagder, Thanks so much for taking a look as well. As far as I can see userpwd is correctly processed since if you look at the failed request below the authorization header "Basic Y2F0bWFpZDojRFphUkpZcml4S0UqZ0ZZ" is added and this matches the "user:password" combination. But what goes wrong is that the URL is then changed from

"https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations/"

to

"https://catmaid/#liG81Rk9NYi*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations/"

i.e. it is changed from https://<url> to https://<user>/<password>@url. As I noted above this only happens on windows when user / password are supplied as curl options. If I precompute an authorization header identical to the one that is used in the failed request below, everything works fine. I personally suspect the hash at the start of the password as the problem.

> library(curl)
> h3=curl::new_handle()
> h3=handle_setopt(h3, username="catmaid", password="#DZaRJYrixKE*gFY", verbose=TRUE)
> h3=handle_setheaders(h3, .list = list(
+   'X-Authorization' = 'Token ee6e7fdb03a0d35b3a6f499d0f8f610686551d51'))

> res=curl_fetch_memory("https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations", h3)
Error in curl_fetch_memory("https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations",  : 
  Timeout was reached: [catmaid] Resolving timed out after 2000 milliseconds
> curl::curl_echo(h3)

Connecting...
Request Complete!
$method
[1] "GET"

$path
[1] "/"

$query
[1] ""

$content_type
NULL

$body
NULL

$headers
                                                                   accept                                                           accept-encoding 
                                                                    "*/*"                                                           "deflate, gzip" 
                                                            authorization                                                                      host 
                                 "Basic Y2F0bWFpZDojRFphUkpZcml4S0UqZ0ZZ"                                                                 "catmaid" 
                                                               user-agent                                                           x-authorization 
"RStudio Desktop (1.2.5033); R (3.6.2 x86_64-w64-mingw32 x86_64 mingw32)"                          "Token ee6e7fdb03a0d35b3a6f499d0f8f610686551d51'" 

$url
[1] "https://catmaid/#liG81Rk9NYi*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations/"

bagder · 2020-05-14T12:08:31Z

Did you really mean https://catmaid/#liG81Rk9NYi*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations/ ?

That URL has no user credentials provided. That's a fragment after the hostname catmaid and the single slash as path.

There's supposed to be a colon between the user and the password, not a slash.

jefferis · 2020-05-14T12:15:28Z

No I didn’t! It’s what my original request gets turned into on Windows when I supply user and password as curl options! I’m sorry my message may not have been that easy to understand but maybe try re-reading slowly. If not I will try to rephrase again. Thank you again so much for your input, Greg.

…

Sent from my iPhone

On 14 May 2020, at 13:08, Daniel Stenberg ***@***.***> wrote: Did you really mean ***@***.***/catmaid/2/annotations/ ? That URL has no user credentials provided. That's a fragment after the hostname catmaid and the single slash as path. There's supposed to be a colon between the user and the password, not a slash. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

jeroen · 2020-05-14T12:16:14Z

@bagder libcurl is creating that url. These are his inputs:

CURLOPT_URL: "https://fuzo-sthi2.biol.lu.se/catmaid/2/annotations"
CURLOPT_USERPWD: "catmaid:#DZaRJYrixKE*gFY"

But then at the redirect, libcurl messes that up:

< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Thu, 14 May 2020 09:20:26 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Location: /catmaid/2/annotations/
< X-Robots-Tag: noindex, nofollow
< 
* Connection #0 to host fuzo-sthi2.biol.lu.se left intact
* Issue another request to this URL: 'https://catmaid/#DZaRJYrixKE*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations/'
* NTLM-proxy picked AND auth done set, clear picked!
* Could not resolve host: catmaid

bagder · 2020-05-14T12:21:43Z

Gotcha. Digging.

bagder · 2020-05-14T12:23:14Z

I can reproduce this bug with just curl on Linux! Will work on fix.

jeroen · 2020-05-14T12:30:00Z

Oh that's bad news for him :-)

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224

bagder · 2020-05-14T12:42:52Z

Yes, I realize that. I can only apologize and fix the issue. A work-around is possibly to change the password to not use a "hash" (#) or at symbol (@) until running a fixed version.

bagder · 2020-05-14T12:47:40Z

Another work-around is probably to set the credentials in the URL (which requires them to be URL encoded, %23 instead of #):

https://catmaid:%23DZaRJYrixKE*gFY@fuzo-sthi2.biol.lu.se/catmaid/2/annotations

jefferis · 2020-05-14T12:50:47Z

Aha, it is the hash! Thanks so much both of you! Do you think this bug is present in the current version of libcurl but not in an older version that I have on my Mac? Or is there something platform specific?

…

Sent from my iPhone

On 14 May 2020, at 13:43, Daniel Stenberg ***@***.***> wrote: Yes, I realize that. I can only apologize and fix the issue. A work-around is possibly to change the password to not use a "hash" (#) or at symbol (@) until running a fixed version. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

bagder · 2020-05-14T12:53:56Z

If the mac version is older than 7.62.0 then I think that's the explanation. I think this is a regression that came with 7.62.0 (which brought the refreshed URL parser internally in curl) or possibly some version later and as I could reproduce it now in my dev version it is still present in 7.70.0.

jefferis · 2020-05-14T12:56:12Z

Yes mac is older

$ curl --version
curl 7.54.0 (x86_64-apple-darwin18.0) libcurl/7.54.0 LibreSSL/2.6.5 zlib/1.2.11 nghttp2/1.24.1
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz HTTP2 UnixSockets HTTPS-proxy
(base) Gregs-MBP-2:catmaid jefferis$

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224 Closes #5400

bagder added a commit to curl/curl that referenced this issue May 14, 2020

url: make the updated credentials URL-encoded in the URL

b670f22

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224

bagder mentioned this issue May 14, 2020

url: make the updated credentials URL-encoded in the URL curl/curl#5400

Closed

bagder added a commit to curl/curl that referenced this issue May 14, 2020

url: make the updated credentials URL-encoded in the URL

600a8cd

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224 Closes #5400

cmb69 added a commit to winlibs/cURL that referenced this issue Jul 7, 2020

url: make the updated credentials URL-encoded in the URL

e56fef9

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224 Closes #5400

cmb69 added a commit to winlibs/cURL that referenced this issue Jul 7, 2020

url: make the updated credentials URL-encoded in the URL

bcd35b1

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224 Closes #5400

cmb69 pushed a commit to winlibs/cURL that referenced this issue Jul 7, 2020

url: make the updated credentials URL-encoded in the URL

36c5846

Found-by: Gregory Jefferis Reported-by: Jeroen Ooms Added test 1168 to verify. Bug spotted when doing a redirect. Bug: jeroen/curl#224 Closes #5400

jay mentioned this issue Sep 12, 2020

Basic auth with Non-URL encoded @ sign in password in user argument breaks when curl follows relative redirects. curl/curl#5949

Closed

OscarLane mentioned this issue Feb 5, 2021

Unable to set default proxy for curl requests on Windows 10 #237

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows specific failure with basic auth using userpwd curl option but not header #224

Windows specific failure with basic auth using userpwd curl option but not header #224

jefferis commented May 14, 2020 •

edited

Loading

jeroen commented May 14, 2020

jefferis commented May 14, 2020 via email

jeroen commented May 14, 2020

jefferis commented May 14, 2020

jefferis commented May 14, 2020

jeroen commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020 via email

jeroen commented May 14, 2020 •

edited

Loading

bagder commented May 14, 2020

bagder commented May 14, 2020

jeroen commented May 14, 2020

bagder commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020 via email

bagder commented May 14, 2020

jefferis commented May 14, 2020 •

edited

Loading

Windows specific failure with basic auth using userpwd curl option but not header #224

Windows specific failure with basic auth using userpwd curl option but not header #224

Comments

jefferis commented May 14, 2020 • edited Loading

jeroen commented May 14, 2020

jefferis commented May 14, 2020 via email

jeroen commented May 14, 2020

jefferis commented May 14, 2020

jefferis commented May 14, 2020

jeroen commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020 via email

jeroen commented May 14, 2020 • edited Loading

bagder commented May 14, 2020

bagder commented May 14, 2020

jeroen commented May 14, 2020

bagder commented May 14, 2020

bagder commented May 14, 2020

jefferis commented May 14, 2020 via email

bagder commented May 14, 2020

jefferis commented May 14, 2020 • edited Loading

jefferis commented May 14, 2020 •

edited

Loading

jeroen commented May 14, 2020 •

edited

Loading

jefferis commented May 14, 2020 •

edited

Loading