winbuild: http digest authentication not working with SHA-256

[ganeshkamath89]

As per RFC7616: https://tools.ietf.org/html/rfc7616
When I set algorithm as MD5 or MD5-sess, Digest authentication is working succesfully. Eg HTTP header:
MD5:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="MD5"

MD5-sess:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="MD5-sess"

I tried setting algorithm as SHA-256 for HTTP Digest authentication as per RFC7616, but with this Digest authentication is failing.
Eg http header:
SHA-256:
username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="SHA-256"

For security purpose, in the example I have changed some strings.

Can someone confirm that it is an issue with Curl or the syntax of HTTP header? Also, does Curl comply with RFC-7616, any plans to support this soon?

[bagder]

So what happens when the server asks for sha-256? What TLS backend are you using? What curl version is this ? What Windows version? Did you build curl yourself or is this the binary we provide?

[ganeshkamath89]

Okay

TLS Backend: OpenSSL 1.1.1h
Curl Version: 0.73
Windows version: Windows 10, Windows Server 2016

We build Curl from source.

When server asks for sha-256, there is no response from our client which uses curl.

Since we build Curl with nmake and not as a VS project, currently the issues is not debugged yet. But we are looking for a fix at the earliest.

[bagder]

Curl Version: 0.73

I presume you mean 7.73.0 ?

there is no response from our client which uses curl

Are you saying that curl doesn't send the follow-up request with authentication provided after the server send back a response header with a sha-256 Digest in it? But it doesn't return any error?

Can you provide a set of headers that should've worked but didn't so that we can try in our end? We have several test cases using sha-256 that keep working...

But we are looking for a fix at the earliest.

Don't we all?

[jay]

If you are using curl please give us the curl -V or libcurl please give us the curl_version()

  curl_version_info_data *info = curl_version_info(CURLVERSION_NOW);
  printf("libcurl SSPI support is %s\n", ((info->features & CURL_VERSION_SSPI) ? "on" : "off"));
  printf("libcurl UNICODE support is %s\n", ((info->features & CURL_VERSION_UNICODE) ? "on" : "off"));
  printf("%s\n", curl_version());

As @bagder said there are already tests that use sha256 and pass.

[ganeshkamath89]

@jay , I am getting this output:

libcurl SSPI support is on
libcurl UNICODE support is off
libcurl/7.73.0 OpenSSL/1.1.1h WinIDN

I have tried both capital SHA-256 and smallcase sha-256, with and without the hiphen.

[bagder]

It should use a hiphen:

curl/lib/vauth/digest.c

Lines 617 to 618 in a0f0c71

	else if(strcasecompare(content, "SHA-256"))
	digest->algo = CURLDIGESTALGO_SHA256;

[ganeshkamath89]

@bagder , thank you for sharing the code path.

In the function: Curl_auth_create_digest_md5_message, I see that the computation of Hash is done like this:

Curl_MD5_update(ctxt, (const unsigned char *) userp,
                 curlx_uztoui(strlen(userp)));
 Curl_MD5_update(ctxt, (const unsigned char *) ":", 1);
 Curl_MD5_update(ctxt, (const unsigned char *) realm,
                 curlx_uztoui(strlen(realm)));
 Curl_MD5_update(ctxt, (const unsigned char *) ":", 1);
 Curl_MD5_update(ctxt, (const unsigned char *) passwdp,
                 curlx_uztoui(strlen(passwdp)));
 Curl_MD5_final(ctxt, digest);

But during the computation of Hash, the algorithm sent by the server is not being used, but it appears to be using MD5.

As per Wikipedia: https://en.wikipedia.org/wiki/Digest_access_authentication. This is equivalent of
HA1 = MD5(username:realm:password)

where it also says: "In September 2015, RFC 7616 replaced RFC 2617 by adding 4 new algorithms: "SHA-256", "SHA-256-sess", "SHA-512" and "SHA-512-sess". The encoding is equivalent to "MD5" and "MD5-sess" algorithms, with MD5 hashing function replaced with SHA-256 and SHA-512."

I am just checking with you, if the Curl_MD5_update and Curl_MD5_final internally always uses MD5 or does it replaces SHA-256 when SHA-256 is mentioned in the algorithm attribute.

Or, is there somewhere an eqivalent Curl_auth_create_digest_sha256_message function somewhere else which specifically handles SHA-256

Because when SHA-256 is passed in the algorithm field, the HA1 computation should be:
HA1 = SHA-256(username:realm:password)

[jay]

The most basic sha256 digest test we have is test2061. The code in libcurl that calculates digest depends on your SSL backend (and on Windows whether SSPI is enabled). curl digest tests aren't run with builds of curl that support SSPI because their digest strings are formatted slightly different.

SSPI builds of curl use the Windows OS functions to generate the digest string. It seems that Windows doesn't support sha-256 digest. InitializeSecurityContext returns error SEC_E_QOP_NOT_SUPPORTED which is not a documented error code for that function.

curl/lib/vauth/digest_sspi.c

Lines 601 to 625 in e052859

	/* Generate our response message */
	status = s_pSecFn->InitializeSecurityContext(&credentials, NULL,
	spn,
	ISC_REQ_USE_HTTP_STYLE, 0, 0,
	&chlg_desc, 0,
	digest->http_context,
	&resp_desc, &attrs, &expiry);
	curlx_unicodefree(spn);
	if(status == SEC_I_COMPLETE_NEEDED ||
	status == SEC_I_COMPLETE_AND_CONTINUE)
	s_pSecFn->CompleteAuthToken(&credentials, &resp_desc);
	else if(status != SEC_E_OK && status != SEC_I_CONTINUE_NEEDED) {
	s_pSecFn->FreeCredentialsHandle(&credentials);
	Curl_sspi_free_identity(p_identity);
	free(output_token);
	Curl_safefree(digest->http_context);
	if(status == SEC_E_INSUFFICIENT_MEMORY)
	return CURLE_OUT_OF_MEMORY;
	return CURLE_AUTH_ERROR;
	}

cygwin prompt run a server:

cat > resp << EOF
HTTP/1.1 401 Authorization Required
WWW-Authenticate: Digest username="Postman",realm="PostOffice",nonce="5bf1156647e8eb42",uri="/wsman",cnonce="4b67562475544ee9090aff3c6d686eab",nc=00000001,response="99de2a7e6c6c3025fd67d7c9d625e0ee",qop="auth",opaque="661d9eae",algorithm="SHA-256"
Content-Length: 0

EOF

while true; do cat resp | nc -4l localhost 8000; done

windows command prompt run curl:

curl -v -u testuser:testpass --digest "http://127.0.0.1:8000/"

result on Windows 7 & 10: curl: (94) An authentication function returned an error

#6315 adds support for more details on the error

* schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED
(0x8009030A) - The per-message Quality of Protection is not supported by
the security package
* multi_done
* Connection #1 to host 127.0.0.1 left intact
curl: (94) An authentication function returned an error

I can't determine from the documentation whether or not sha256 digest is supported so for now I think we have to assume it isn't. To work around this you would have to build libcurl without USE_WINDOWS_SSPI defined which will cause libcurl to use its own functions to make the digest. To do that with winbuild specify ENABLE_SSPI=no

[ganeshkamath89]

@bagder & @jay

Thank you very much. I disabled SSPI as per your suggestion and got a successful response from Curl.

Request from server:
HTTP/1.1 401 Unauthorized Content-Length: 0 Connection: Keep-Alive Content-Type: application/soap+xml;charset=UTF-8 WWW-Authenticate: Digest realm="Ganesh-Dev", nonce="5bf1156647e8eb43", algorithm="SHA-256", qop="auth", opaque="661d9ebe" WWW-Authenticate: Basic realm="PostOffice"

Response from client:
POST /wsman HTTP/1.1 Host: 192.168.1.1:4467 Authorization: Digest username="Postman", realm="PostOffice", nonce="5bf1156647e8eb43", uri="/wsman", cnonce="YWNjNDA0NGY1MmM1MGViMzE1NWY5YzBhNzMxNWY4MWY+", nc=00000001, qop=auth, response="183295c351c5e2903c7a248630e6e68d3009262738723c1814a0f6f572c27f64", opaque="661d9ebe", algorithm=SHA-256 Accept: */* Content-Type: application/soap+xml;charset=UTF-8 User-Agent: Openwsman Content-Length: 884

I have edited the values for security purpose.

You can close this issue.

[ganeshkamath89]

Today I tried with SHA-512. Now I am getting a different behaviour.

Note: I tested SHA-256 and SHA-256-sess thoroughly and it is working consistantly.

Request from server:

HTTP/1.1 401 Unauthorized 
Content-Length: 0 
Connection: Keep-Alive 
Content-Type: application/soap+xml;charset=UTF-8 
WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe" 
WWW-Authenticate: Basic realm="PostOffice"

Response from client (where I use Curl):

POST /wsman HTTP/1.1 
Host: 192.168.1.1:4467 
Authorization: Digest username="Postman", realm="PostOffice", , nonce="5bf1156647e8eb42", uri="/wsman", response="0a3bff0b14cf48619cf395dc5494085d", algorithm=SHA-512

I am not even getting back qop/opaque/cnonce/nc in the response.

For us SHA-512 is not a current requirement. Just checking in the forum in case I am doing something wrong.

[jay]

Response from client:
POST /wsman HTTP/1.1 Host: 192.168.1.1:4467 Authorization: Digest username="Postman", realm="PostOffice", , nonce="5bf1156647e8eb42", uri="/wsman", response="0a3bff0b14cf48619cf395dc5494085d", algorithm=SHA-512

I am not even getting back qop/opaque/cnonce/nc in the response.

curl doesn't support SHA-512 digest so the auth function returns an error that should cause the authentication header to be ignored. In the case of unsupported digest algorithms in non-SSPI builds the error doesn't bubble up it is just ignored. I can't explain why you're seeing curl respond with SHA-512 which it does not support. It may support SHA-512-256 and SHA-512-256-SESS digest as basically sha256 but not SHA-512.

If you have a way to reproduce curl replying with the SHA-512 algorithm then we can investigate it as a bug. I am not able to reproduce with the server data you gave.

[ganeshkamath89]

@jay ,
What kind of data do you want, because I can't share code?
Give me some time to collect the data.

Also, what is SHA-512-256, how do I hash with this algorithm? because I want to verify the response from server.
for MD5 and sha256, I am using openssl provided algorithms, but I don't know about SHA-512-256.

[jay]

What kind of data do you want, because I can't share code?

I would need server WWW-Authenticate header that I can use to reproduce the issue. I cannot reproduce with the WWW-Authenticate you have already given.

Also, what is SHA-512-256, how do I hash with this algorithm? because I want to verify the response from server.
for MD5 and sha256, I am using openssl provided algorithms, but I don't know about SHA-512-256.

SHA-512-256 as far as I can tell is a truncated version of SHA-512. curl uses the sha256 hash function for SHA-512-256. Other than that I don't know much about it. If your server supports SHA-512-256 and requires that algorithm for digest authentication then I think curl should be able to work with that.

[ganeshkamath89]

@jay, this itself is my full WWW authenticate header for Digest.

WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"

I have reformatted my previous answer so that it is more readable.

Your suggestion to use Curl in the server is a good idea. I will explore it after my current tasks are complete.

[jay]

WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"

I can't reproduce with that header. The algorithm is properly ignored and you can see in verbose mode it says Authentication problem. Ignoring this.

* STATE: INIT => CONNECT handle 0x447b08; line 1654 (connection #-5000)
* Added connection 0. The cache now contains 1 members
* family0 == v4, family1 == v6
*   Trying 127.0.0.1:8000...
* STATE: CONNECT => WAITCONNECT handle 0x447b08; line 1715 (connection #0)
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
* STATE: WAITCONNECT => SENDPROTOCONNECT handle 0x447b08; line 1848 (connection #0)
* Marked for [keep alive]: HTTP default
* STATE: SENDPROTOCONNECT => DO handle 0x447b08; line 1869 (connection #0)
* Server auth using Digest with user 'testuser'
> GET / HTTP/1.1
> Host: 127.0.0.1:8000
> User-Agent: curl/7.74.1-DEV
> Accept: */*
>
* STATE: DO => DO_DONE handle 0x447b08; line 1940 (connection #0)
* STATE: DO_DONE => PERFORM handle 0x447b08; line 2061 (connection #0)
* Mark bundle as not supporting multiuse
* HTTP 1.1 or later with persistent connection
< HTTP/1.1 401 Authorization Required
* Authentication problem. Ignoring this.
< WWW-Authenticate: Digest realm="PostOffice", nonce="5bf1156647e8eb43", algorithm="SHA-512", qop="auth", opaque="661d9ebe"
< Content-Length: 0
<
* STATE: PERFORM => DONE handle 0x447b08; line 2251 (connection #0)
* multi_done
* Connection #0 to host 127.0.0.1 left intact
* Expire cleared (transfer 0x447b08)

Your suggestion to use Curl in the server is a good idea. I will explore it after my current tasks are complete.

To be clear curl can't be used as a server, what I was saying is that curl as a client (either the curl tool curl.exe or your program using libcurl) should be able to handle SHA-512-256 authenticate from your server.

[ganeshkamath89]

My curl client looks somewhat like this example: https://github.com/Openwsman/openwsman/blob/master/src/lib/wsman-curl-client-transport.c

Hope the values set in this file help you in reproducing the issue.

Also, I have verified the SHA-512-256 functionality. I found out that SHA-512-256, does the same work as SHA-256.