Skip to content

winbuild: http digest authentication not working with SHA-256 #6302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ganeshkamath89 opened this issue Dec 11, 2020 · 16 comments
Closed

winbuild: http digest authentication not working with SHA-256 #6302

ganeshkamath89 opened this issue Dec 11, 2020 · 16 comments

Comments

@ganeshkamath89
Copy link

ganeshkamath89 commented Dec 11, 2020

As per RFC7616: https://tools.ietf.org/html/rfc7616
When I set algorithm as MD5 or MD5-sess, Digest authentication is working succesfully. Eg HTTP header:
MD5:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="MD5"

MD5-sess:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="MD5-sess"

I tried setting algorithm as SHA-256 for HTTP Digest authentication as per RFC7616, but with this Digest authentication is failing.
Eg http header:
SHA-256:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="SHA-256"

For security purpose, in the example I have changed some strings.

Can someone confirm that it is an issue with Curl or the syntax of HTTP header? Also, does Curl comply with RFC-7616, any plans to support this soon?

@bagder
Copy link
Member

bagder commented Dec 11, 2020

So what happens when the server asks for sha-256? What TLS backend are you using? What curl version is this ? What Windows version? Did you build curl yourself or is this the binary we provide?

@bagder bagder changed the title winbuild: http digest authentication not working with algorithm set as SHA256 in header but working with less secure MD5 and MD5-sess winbuild: http digest authentication not working with SHA-256 Dec 11, 2020
@ganeshkamath89
Copy link
Author

Okay

TLS Backend: OpenSSL 1.1.1h
Curl Version: 0.73
Windows version: Windows 10, Windows Server 2016

We build Curl from source.

When server asks for sha-256, there is no response from our client which uses curl.

Since we build Curl with nmake and not as a VS project, currently the issues is not debugged yet. But we are looking for a fix at the earliest.

@bagder
Copy link
Member

bagder commented Dec 11, 2020

Curl Version: 0.73

I presume you mean 7.73.0 ?

there is no response from our client which uses curl

Are you saying that curl doesn't send the follow-up request with authentication provided after the server send back a response header with a sha-256 Digest in it? But it doesn't return any error?

Can you provide a set of headers that should've worked but didn't so that we can try in our end? We have several test cases using sha-256 that keep working...

But we are looking for a fix at the earliest.

Don't we all?

@jay
Copy link
Member

jay commented Dec 11, 2020

If you are using curl please give us the curl -V or libcurl please give us the curl_version()

  curl_version_info_data *info = curl_version_info(CURLVERSION_NOW);
  printf("libcurl SSPI support is %s\n", ((info->features & CURL_VERSION_SSPI) ? "on" : "off"));
  printf("libcurl UNICODE support is %s\n", ((info->features & CURL_VERSION_UNICODE) ? "on" : "off"));
  printf("%s\n", curl_version());

As @bagder said there are already tests that use sha256 and pass.

@ganeshkamath89
Copy link
Author

@jay , I am getting this output:

libcurl SSPI support is on
libcurl UNICODE support is off
libcurl/7.73.0 OpenSSL/1.1.1h WinIDN

I have tried both capital SHA-256 and smallcase sha-256, with and without the hiphen.

@bagder
Copy link
Member

bagder commented Dec 12, 2020

It should use a hiphen:

curl/lib/vauth/digest.c

Lines 617 to 618 in a0f0c71

else if(strcasecompare(content, "SHA-256"))
digest->algo = CURLDIGESTALGO_SHA256;

@ganeshkamath89
Copy link
Author

ganeshkamath89 commented Dec 13, 2020

@bagder , thank you for sharing the code path.

In the function: Curl_auth_create_digest_md5_message, I see that the computation of Hash is done like this:

Curl_MD5_update(ctxt, (const unsigned char *) userp,
                 curlx_uztoui(strlen(userp)));
 Curl_MD5_update(ctxt, (const unsigned char *) ":", 1);
 Curl_MD5_update(ctxt, (const unsigned char *) realm,
                 curlx_uztoui(strlen(realm)));
 Curl_MD5_update(ctxt, (const unsigned char *) ":", 1);
 Curl_MD5_update(ctxt, (const unsigned char *) passwdp,
                 curlx_uztoui(strlen(passwdp)));
 Curl_MD5_final(ctxt, digest);

But during the computation of Hash, the algorithm sent by the server is not being used, but it appears to be using MD5.

As per Wikipedia: https://en.wikipedia.org/wiki/Digest_access_authentication. This is equivalent of
HA1 = MD5(username:realm:password)

where it also says: "In September 2015, RFC 7616 replaced RFC 2617 by adding 4 new algorithms: "SHA-256", "SHA-256-sess", "SHA-512" and "SHA-512-sess". The encoding is equivalent to "MD5" and "MD5-sess" algorithms, with MD5 hashing function replaced with SHA-256 and SHA-512."

I am just checking with you, if the Curl_MD5_update and Curl_MD5_final internally always uses MD5 or does it replaces SHA-256 when SHA-256 is mentioned in the algorithm attribute.

Or, is there somewhere an eqivalent Curl_auth_create_digest_sha256_message function somewhere else which specifically handles SHA-256

Because when SHA-256 is passed in the algorithm field, the HA1 computation should be:
HA1 = SHA-256(username:realm:password)

jay added a commit to jay/curl that referenced this issue Dec 13, 2020
The error is shown with infof rather than failf so that the user will
see the extended error message information only in verbose mode, and
will still see the standard CURLE_AUTH_ERROR message. For example:

---

* schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED
(0x8009030A) - The per-message Quality of Protection is not supported by
the security package
* multi_done
* Connection curl#1 to host 127.0.0.1 left intact
curl: (94) An authentication function returned an error

---

Ref: curl#6302

Closes #xxxx
@jay
Copy link
Member

jay commented Dec 13, 2020

The most basic sha256 digest test we have is test2061. The code in libcurl that calculates digest depends on your SSL backend (and on Windows whether SSPI is enabled). curl digest tests aren't run with builds of curl that support SSPI because their digest strings are formatted slightly different.

SSPI builds of curl use the Windows OS functions to generate the digest string. It seems that Windows doesn't support sha-256 digest. InitializeSecurityContext returns error SEC_E_QOP_NOT_SUPPORTED which is not a documented error code for that function.

/* Generate our response message */
status = s_pSecFn->InitializeSecurityContext(&credentials, NULL,
spn,
ISC_REQ_USE_HTTP_STYLE, 0, 0,
&chlg_desc, 0,
digest->http_context,
&resp_desc, &attrs, &expiry);
curlx_unicodefree(spn);
if(status == SEC_I_COMPLETE_NEEDED ||
status == SEC_I_COMPLETE_AND_CONTINUE)
s_pSecFn->CompleteAuthToken(&credentials, &resp_desc);
else if(status != SEC_E_OK && status != SEC_I_CONTINUE_NEEDED) {
s_pSecFn->FreeCredentialsHandle(&credentials);
Curl_sspi_free_identity(p_identity);
free(output_token);
Curl_safefree(digest->http_context);
if(status == SEC_E_INSUFFICIENT_MEMORY)
return CURLE_OUT_OF_MEMORY;
return CURLE_AUTH_ERROR;
}

cygwin prompt run a server:

cat > resp << EOF
HTTP/1.1 401 Authorization Required
WWW-Authenticate: Digest username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="SHA-256"
Content-Length: 0

EOF

while true; do cat resp | nc -4l localhost 8000; done

windows command prompt run curl:

curl -v -u testuser:testpass --digest "http://127.0.0.1:8000/"

result on Windows 7 & 10: curl: (94) An authentication function returned an error

#6315 adds support for more details on the error

* schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED
(0x8009030A) - The per-message Quality of Protection is not supported by
the security package
* multi_done
* Connection #1 to host 127.0.0.1 left intact
curl: (94) An authentication function returned an error

I can't determine from the documentation whether or not sha256 digest is supported so for now I think we have to assume it isn't. To work around this you would have to build libcurl without USE_WINDOWS_SSPI defined which will cause libcurl to use its own functions to make the digest. To do that with winbuild specify ENABLE_SSPI=no

@ganeshkamath89
Copy link
Author

ganeshkamath89 commented Dec 13, 2020

@bagder & @jay

Thank you very much. I disabled SSPI as per your suggestion and got a successful response from Curl.

Request from server:
HTTP/1.1 401 Unauthorized Content-Length: 0 Connection: Keep-Alive Content-Type: application/soap+xml;charset=UTF-8 WWW-Authenticate: Digest realm="Ganesh-Dev", nonce="5bf1156647e8eb43", algorithm="SHA-256", qop="auth", opaque="661d9ebe" WWW-Authenticate: Basic realm="PostOffice"

Response from client:
POST /wsman HTTP/1.1 Host: 192.168.1.1:4467 Authorization: Digest username="Postman", realm="PostOffice", nonce="5bf1156647e8eb43", uri="/wsman", cnonce="YWNjNDA0NGY1MmM1MGViMzE1NWY5YzBhNzMxNWY4MWY+", nc=00000001, qop=auth, response="183295c351c5e2903c7a248630e6e68d3009262738723c1814a0f6f572c27f64", opaque="661d9ebe", algorithm=SHA-256 Accept: */* Content-Type: application/soap+xml;charset=UTF-8 User-Agent: Openwsman Content-Length: 884

I have edited the values for security purpose.

You can close this issue.

@bagder bagder closed this as completed Dec 13, 2020
@ganeshkamath89
Copy link
Author

ganeshkamath89 commented Dec 14, 2020

Today I tried with SHA-512. Now I am getting a different behaviour.

Note: I tested SHA-256 and SHA-256-sess thoroughly and it is working consistantly.

Request from server:

HTTP/1.1 401 Unauthorized 
Content-Length: 0 
Connection: Keep-Alive 
Content-Type: application/soap+xml;charset=UTF-8 
WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe" 
WWW-Authenticate: Basic realm="PostOffice"

Response from client (where I use Curl):

POST /wsman HTTP/1.1 
Host: 192.168.1.1:4467 
Authorization: Digest username="Postman", realm="PostOffice", , nonce="5bf1156647e8eb42", uri="/wsman", response="0a3bff0b14cf48619cf395dc5494085d", algorithm=SHA-512

I am not even getting back qop/opaque/cnonce/nc in the response.

For us SHA-512 is not a current requirement. Just checking in the forum in case I am doing something wrong.

jay added a commit that referenced this issue Dec 14, 2020
The error is shown with infof rather than failf so that the user will
see the extended error message information only in verbose mode, and
will still see the standard CURLE_AUTH_ERROR message. For example:

---

* schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED
(0x8009030A) - The per-message Quality of Protection is not supported by
the security package
* multi_done
* Connection #1 to host 127.0.0.1 left intact
curl: (94) An authentication function returned an error

---

Ref: #6302

Closes #6315
@jay
Copy link
Member

jay commented Dec 14, 2020

Response from client:
POST /wsman HTTP/1.1 Host: 192.168.1.1:4467 Authorization: Digest username="Postman", realm="PostOffice", , nonce="5bf1156647e8eb42", uri="/wsman", response="0a3bff0b14cf48619cf395dc5494085d", algorithm=SHA-512

I am not even getting back qop/opaque/cnonce/nc in the response.

curl doesn't support SHA-512 digest so the auth function returns an error that should cause the authentication header to be ignored. In the case of unsupported digest algorithms in non-SSPI builds the error doesn't bubble up it is just ignored. I can't explain why you're seeing curl respond with SHA-512 which it does not support. It may support SHA-512-256 and SHA-512-256-SESS digest as basically sha256 but not SHA-512.

If you have a way to reproduce curl replying with the SHA-512 algorithm then we can investigate it as a bug. I am not able to reproduce with the server data you gave.

@ganeshkamath89
Copy link
Author

@jay ,
What kind of data do you want, because I can't share code?
Give me some time to collect the data.

Also, what is SHA-512-256, how do I hash with this algorithm? because I want to verify the response from server.
for MD5 and sha256, I am using openssl provided algorithms, but I don't know about SHA-512-256.

@jay
Copy link
Member

jay commented Dec 15, 2020

What kind of data do you want, because I can't share code?

I would need server WWW-Authenticate header that I can use to reproduce the issue. I cannot reproduce with the WWW-Authenticate you have already given.

Also, what is SHA-512-256, how do I hash with this algorithm? because I want to verify the response from server.
for MD5 and sha256, I am using openssl provided algorithms, but I don't know about SHA-512-256.

SHA-512-256 as far as I can tell is a truncated version of SHA-512. curl uses the sha256 hash function for SHA-512-256. Other than that I don't know much about it. If your server supports SHA-512-256 and requires that algorithm for digest authentication then I think curl should be able to work with that.

@ganeshkamath89
Copy link
Author

@jay, this itself is my full WWW authenticate header for Digest.

WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"

I have reformatted my previous answer so that it is more readable.

Your suggestion to use Curl in the server is a good idea. I will explore it after my current tasks are complete.

@jay
Copy link
Member

jay commented Dec 15, 2020

WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"

I can't reproduce with that header. The algorithm is properly ignored and you can see in verbose mode it says Authentication problem. Ignoring this.

* STATE: INIT => CONNECT handle 0x447b08; line 1654 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* family0 == v4, family1 == v6
*   Trying 127.0.0.1:8000...
* STATE: CONNECT => WAITCONNECT handle 0x447b08; line 1715 (connection #0)
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x447b08; line 1848 (connection #0)
* Marked for [keep alive]: HTTP default
* STATE: SENDPROTOCONNECT => DO handle 0x447b08; line 1869 (connection #0)
* Server auth using Digest with user 'testuser'
> GET / HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.74.1-DEV
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x447b08; line 1940 (connection #0)
* STATE: DO_DONE => PERFORM handle 0x447b08; line 2061 (connection #0)
* Mark bundle as not supporting multiuse
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 401 Authorization Required
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"
< Content-Length: 0
<
* STATE: PERFORM => DONE handle 0x447b08; line 2251 (connection #0)
* multi_done
* Connection #0 to host 127.0.0.1 left intact
* Expire cleared (transfer 0x447b08)

Your suggestion to use Curl in the server is a good idea. I will explore it after my current tasks are complete.

To be clear curl can't be used as a server, what I was saying is that curl as a client (either the curl tool curl.exe or your program using libcurl) should be able to handle SHA-512-256 authenticate from your server.

@ganeshkamath89
Copy link
Author

ganeshkamath89 commented Dec 16, 2020

My curl client looks somewhat like this example: https://github.com/Openwsman/openwsman/blob/master/src/lib/wsman-curl-client-transport.c

Hope the values set in this file help you in reproducing the issue.

Also, I have verified the SHA-512-256 functionality. I found out that SHA-512-256, does the same work as SHA-256.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

3 participants