ssl certificate processing logic error

[pedoc]

Description

The behavior on core and .NET Framework is inconsistent, and the SSL certificate cannot be handled correctly, resulting in the failure of business code

Reproduction Steps

sample code

using System.Net;
using System.Net.Security;

ServicePointManager.ServerCertificateValidationCallback =
           new RemoteCertificateValidationCallback(
                delegate
                {

                    return true;
                }
            );
var wb = new WebClient();
wb.DownloadFile("https://pubuserqiniu.paperol.cn/181807649_54_q11_1665653480Ytkkdt.docx?attname=55_11_%e6%b0%b4%e5%ba%93%e7%94%9f%e6%80%81%e9%b1%bc%e7%b2%be%e5%87%86%e6%8d%95%e6%8d%9e%e6%97%a0%e6%8a%97%e8%bf%90%e8%be%9310.13.docx&e=1674795131&token=-kY3jr8KMC7l3KkIN3OcIs8Q4s40OfGgUHr1Rg4D:3QdoI-6iu-Bc6ebPLVzbvvQhC64=", "test.docx");

When the above code is executed in all netcore versions, an error will occur and the file cannot be downloaded.(even if i ignore certificate error in ServerCertificateValidationCallback，Of course, this also includes HttpClient).

.net 7.0.200 output

But he works fine on .NET Framework

Expected behavior

Able to download files normally

Actual behavior

not working

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

Note that this is not our server and we have no control over server certificates

[ghost]

Tagging subscribers to this area: @dotnet/ncl, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

The behavior on core and .NET Framework is inconsistent, and the SSL certificate cannot be handled correctly, resulting in the failure of business code

Reproduction Steps

sample code

using System.Net;
using System.Net.Security;

ServicePointManager.ServerCertificateValidationCallback =
           new RemoteCertificateValidationCallback(
                delegate
                {

                    return true;
                }
            );
var wb = new WebClient();
wb.DownloadFile("https://pubuserqiniu.paperol.cn/181807649_54_q11_1665653480Ytkkdt.docx?attname=55_11_%e6%b0%b4%e5%ba%93%e7%94%9f%e6%80%81%e9%b1%bc%e7%b2%be%e5%87%86%e6%8d%95%e6%8d%9e%e6%97%a0%e6%8a%97%e8%bf%90%e8%be%9310.13.docx&e=1674795131&token=-kY3jr8KMC7l3KkIN3OcIs8Q4s40OfGgUHr1Rg4D:3QdoI-6iu-Bc6ebPLVzbvvQhC64=", "test.docx");

When the above code is executed in all netcore versions, an error will occur and the file cannot be downloaded.(even if i ignore certificate error in ServerCertificateValidationCallback，Of course, this also includes HttpClient).

.net 7.0.200 output

But he works fine on .NET Framework

Expected behavior

Able to download files normally

Actual behavior

not working

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

Note that this is not our server and we have no control over server certificates

Author:	pedoc
Assignees:	-
Labels:	area-System.Net.Security
Milestone:	-

[wfurt]

Did you actually try this with HttpClient @pedoc? For that you need to set ServerCertificateCustomValidationCallback on handler.
ServicePointManager is obsolete in .NET Core and has minimal functionality for legacy. The WebClient should probably still work and we should investigate.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Description

The behavior on core and .NET Framework is inconsistent, and the SSL certificate cannot be handled correctly, resulting in the failure of business code

Reproduction Steps

sample code

using System.Net;
using System.Net.Security;

ServicePointManager.ServerCertificateValidationCallback =
           new RemoteCertificateValidationCallback(
                delegate
                {

                    return true;
                }
            );
var wb = new WebClient();
wb.DownloadFile("https://pubuserqiniu.paperol.cn/181807649_54_q11_1665653480Ytkkdt.docx?attname=55_11_%e6%b0%b4%e5%ba%93%e7%94%9f%e6%80%81%e9%b1%bc%e7%b2%be%e5%87%86%e6%8d%95%e6%8d%9e%e6%97%a0%e6%8a%97%e8%bf%90%e8%be%9310.13.docx&e=1674795131&token=-kY3jr8KMC7l3KkIN3OcIs8Q4s40OfGgUHr1Rg4D:3QdoI-6iu-Bc6ebPLVzbvvQhC64=", "test.docx");

When the above code is executed in all netcore versions, an error will occur and the file cannot be downloaded.(even if i ignore certificate error in ServerCertificateValidationCallback，Of course, this also includes HttpClient).

.net 7.0.200 output

But he works fine on .NET Framework

Expected behavior

Able to download files normally

Actual behavior

not working

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

Note that this is not our server and we have no control over server certificates

Author:	pedoc
Assignees:	-
Labels:	area-System.Net.Http, tenet-compatibility, untriaged
Milestone:	-

[vcsjones]

SSL certificate cannot be handled correctly

I don't see any evidence that this is an SSL/TLS issue. According the to screenshot, the server responded with an HTTP 403, which indicates a TLS session was established.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[pedoc]

Did you actually try this with HttpClient @pedoc? For that you need to set ServerCertificateCustomValidationCallback on handler.

Yes, I tried HttpClient, problem also exists.

code

using System.Net;
using System.Net.Http;
var addr = "https://pubuserqiniu.paperol.cn/181807649_54_q11_1665653480Ytkkdt.docx?attname=55_11_%e6%b0%b4%e5%ba%93%e7%94%9f%e6%80%81%e9%b1%bc%e7%b2%be%e5%87%86%e6%8d%95%e6%8d%9e%e6%97%a0%e6%8a%97%e8%bf%90%e8%be%9310.13.docx&e=1674795131&token=-kY3jr8KMC7l3KkIN3OcIs8Q4s40OfGgUHr1Rg4D:3QdoI-6iu-Bc6ebPLVzbvvQhC64=";
ServicePointManager.ServerCertificateValidationCallback +=
    (sender, cert, chain, sslPolicyErrors) => true;
var handler = new HttpClientHandler();
handler.ClientCertificateOptions = ClientCertificateOption.Manual;
handler.ServerCertificateCustomValidationCallback =
    (httpRequestMessage, cert, cetChain, policyErrors) =>
{
    return true;
};

var h = new HttpClient(handler);
h.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.62");
h.DefaultRequestHeaders.Add("Accept", "*/*");
h.DefaultRequestHeaders.Add("Accept-Encoding", "gzip,deflate");
var r = await h.GetByteArrayAsync(addr);
File.WriteAllBytes("test下载.docx",r);

[pedoc]

I don't see any evidence that this is an SSL/TLS issue. According the to screenshot, the server responded with an HTTP 403, which indicates a TLS session was established.

First of all, when using WebClient, we can see Status=ProtocolError in the return value.
Secondly, the same code is executed normally under Net Framework, except for the inconsistency of SSL-related processing, I can't think of any other possibilities.

As a supplement, even if he is not an SSL problem, it should work just like Net Framework.

I am posting the wireshark capture here, hoping to help analyze this problem.

net7.0.200-wireshark.pcapng
net7.0.200-wireshark.pcapng.zip

netframework-wireshark.pcapng
netframework-wireshark.zip

[wfurt]

I get 403 in both cases - Framework & 7.0.

StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  Connection: keep-alive
  X-Cache-Status: MISS
  Access-Control-Allow-Origin: *
  X-Error-Info: qiniu_center_auth.center_auth.access
  Date: Wed, 30 Nov 2022 03:34:05 GMT
  Server: JSP3/2.0.14
  Content-Length: 554
  Content-Type: text/html
}

7.0.

StatusCode: 403, ReasonPhrase: 'Forbidden', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
  Server: Byte-nginx
  Date: Wed, 30 Nov 2022 03:32:10 GMT
  Connection: keep-alive
  Byte-Error-Code: 003
  Via: cache01.cqct
  x-request-ip: 75.172.16.224
  x-tt-trace-tag: id=5
  Access-Control-Allow-Origin: *
  x-response-cinfo: 75.172.16.224
  x-response-cache: miss
  Content-Type: application/vnd.openxmlformats-officedocument.wordprocessingml.document
  Content-Length: 54
}

This is curious as I would also expect same response. Framework also sends "Connection: Keep-alive" but that should not be needed as it is default for 1.1 and adding it explicitly did not make any difference.

After some fiddling, I added handler.SslProtocols = System.Security.Authentication.SslProtocols.Tls12; and 7.0 gets response served by Server: JSP3/2.0.14. It is still 403 error but there seems to be some front-end balancer incompatible with TLS 1.3. I agree with @vcsjones , this is not handshake problem, the capture shows full handshake but the site is broken afterwards.

[pedoc]

@wfurt thanks for your investigation

I get 403 in both cases - Framework & 7.0.

After seeing your feedback, I used VS to test and found that, as you said, the result is 403 on both Framework and 7.0.

When I asked this question, I used roslynpad for testing (not in VS). When run with roslynpad, he does download the file normally.

running in VS：

running in roslynpad：

This is the code comparison after decompiling the generated exe respectively：

I'm currently very confused why there are different results, their runtime versions are the same. Perhaps there is some magic in the RoslynPad?

For reference and comparison, I use Python to download locally, and the Python code is as follows

import urllib.request

url='https://pubuserqiniu.paperol.cn/181807649_54_q11_1665653480Ytkkdt.docx?attname=55_11_%e6%b0%b4%e5%ba%93%e7%94%9f%e6%80%81%e9%b1%bc%e7%b2%be%e5%87%86%e6%8d%95%e6%8d%9e%e6%97%a0%e6%8a%97%e8%bf%90%e8%be%9310.13.docx&e=1674795131&token=-kY3jr8KMC7l3KkIN3OcIs8Q4s40OfGgUHr1Rg4D:3QdoI-6iu-Bc6ebPLVzbvvQhC64='
urllib.request.urlretrieve(url,'test-py.docx')

simple but work fine.

Maybe we need to further investigate whether there are any hidden dangers inside .net?

[pedoc]

I compared the wireshark capture packets again, and found that .net7 tried to use TLS1.3 from the very beginning, and when negotiating to upgrade the protocol, the server refused.
When working normally, the interaction always uses TLS1.0 (even if you forcefully specify System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls, it has no effect)

[wfurt]

right. The danger IMHO lives in the site that fails to respond to valid HTTP requests in consistent way.
Are you testing this with HttpClient or the original WebClient?

[pedoc]

Are you testing this with HttpClient or the original WebClient?

both.

When I use HttpClient and forcefully specify handler.SslProtocols =System.Security.Authentication.SslProtocols.Tls;,
it seems that TLS1.0 is indeed used to shake hands with the server, but due to unknown reasons in the subsequent process Still failed.

[pedoc]

The danger IMHO lives in the site that fails to respond to valid HTTP requests in consistent way.

Yes, but this is someone else's server and we have no control over it. But now we need to download documents from it. Now failing on .net for unknown reasons and having to move us to python, I don't think this is normal.

[wfurt]

Use SslProtocol.Tls12. And yes, it is not "normal" but I still feel the site is to blame. So far, I did not see any sign that .NET would fail to follow the RFC.

[pedoc]

After some more tries, I found that it works fine on Framework 4.0 (includ in VS).

This is probably not a very common problem (although we have), so feel free to close this issue.

And I don't really think it will be fixed 😄 .