wget gives certificate error for dotnet.microsoft.com

[tmds]

On Fedora 34:

$ wget https://dot.net/v1/dotnet-install.sh
--2021-10-21 14:14:23--  https://dot.net/v1/dotnet-install.sh
Resolving dot.net (dot.net)... 40.112.72.205, 40.113.200.201, 104.215.148.63, ...
Connecting to dot.net (dot.net)|40.112.72.205|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh [following]
--2021-10-21 14:14:23--  https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh
Resolving dotnet.microsoft.com (dotnet.microsoft.com)... 13.107.213.67, 13.107.246.67, 2620:1ec:46::67, ...
Connecting to dotnet.microsoft.com (dotnet.microsoft.com)|13.107.213.67|:443... connected.
ERROR: The certificate of ‘dotnet.microsoft.com’ is not trusted.

cc @leecow

[steveharter]

Should this be transferred to dotnet/install-scripts?

[tmds]

The issue is with the certificate for dotnet.microsoft.com. It's not specific to the install scripts.

[steveharter]

cc @dagood @marek-safar for awareness.

[dagood]

It doesn't repro for me on Fedora 33 but does repro exactly the same (with slightly different IPs) in a clean Fedora 34 container. Command for easy repro:

$ docker run -it --rm fedora:34 bash -c 'dnf install -y wget; wget https://dot.net/v1/dotnet-install.sh'

This also repros: wget https://dot.net.

@mairaw are you the right person to help with this?

[mairaw]

We had a similar thread before where we couldn't determine why dot.net wasn't working. @ChrisSfanos helped with this before for the dot.net redirection service.

Do you also have issues if you use the dotnet.microsoft.com URL?

[dagood]

Do you also have issues if you use the dotnet.microsoft.com URL?

Yeah, this also repros for me: wget https://dotnet.microsoft.com

--2021-10-21 23:18:34--  https://dotnet.microsoft.com/
Resolving dotnet.microsoft.com (dotnet.microsoft.com)... 13.107.246.70, 13.107.213.70, 2620:1ec:bdf::70, ...
Connecting to dotnet.microsoft.com (dotnet.microsoft.com)|13.107.246.70|:443... connected.
ERROR: The certificate of 'dotnet.microsoft.com' is not trusted.

[mairaw]

Tagging @terrimorton as well

[ChrisSfanos]

So we just did an ECR and rotated the certificates. Do we have a mismatch somewhere? Is the older cert still in use on FrontDoor?

Edit - ok that probably isn't it as Front Door is set to "Latest" for Secret version.

Could this be a weird cache problem? On a machine that repos, are you able to see the certificate it's grabbing? If so, can you share the first 5 characters of the thumbprint?

[dagood]

On a machine that repos, are you able to see the certificate it's grabbing? If so, can you share the first 5 characters of the thumbprint?

Here's what I get:

$ echo -n | openssl s_client -connect dotnet.microsoft.com:443 | openssl x509 -fingerprint
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root G2
verify return:1
depth=1 C = US, O = Microsoft Corporation, CN = Microsoft Azure TLS Issuing CA 06
verify return:1
depth=0 C = US, ST = WA, L = Redmond, O = Microsoft Corporation, CN = dotnet.microsoft.com
verify return:1
DONE
SHA1 Fingerprint=6A:45:41:56:9D:97:33:C4:4E:B5:1E:FB:BA:F9:45:98:8D:4D:CA:E5
-----BEGIN CERTIFICATE-----
MIIIUDCCBjigAwIBAgITMwAcSpNXY6QTeMcfxgAAABxKkzANBgkqhkiG9w0BAQwF
ADBZMQswCQYDVQQGEwJVUzEeMBwGA1UEChMVTWljcm9zb2Z0IENvcnBvcmF0aW9u
[...]

[ChrisSfanos]

Thanks - I checked and that certificate is the latest one (rotated into place on 10/7).

When I visit dotnet.microsoft.com in the browser, that is also the certificate I get the full path is valid. So I admit I'm not sure what is going on in this case yet

[dagood]

The cert appears to be valid, just untrusted. But, I don't know enough about certificate trust to know if that's just nitpicking, or how to investigate further into why it isn't trusted by this distro.

I checked a Fedora 33 Docker image, and it also repros the same as the Fedora 34 Docker image. For me, only my actively used dev box with Fedora 33 didn't repro.

.NET (especially NuGet) has had run-ins with ca-certificates packages on some Linux distros, maybe now it's the website's turn.

[terrimorton]

I ran the SSL Server Test at https://www.ssllabs.com/ssltest/ (and hid the result from the boards).

I am currently digging into the results to see why there are "B" ratings. This may be the key to the underlying issue.

[terrimorton]

Attached is the report
SSL Report for dotnet websites - 2021-10-25.pdf

Some things that stand out:

Certificate 1 - Subject: dotnet.microsoft.com
DNS CAA: No

Certificate 2 - Subject: *.azureedge.net
Alternative names *.azureedge.net *.media.microsoftstream.com *.origin.mediaservices.windows.net *.streaming.mediaservices.windows.net MISMATCH

Trusted No NOT TRUSTED
Mozilla Apple Android Java Windows

General
This server supports TLS 1.0 and TLS 1.1. Grade capped to B.
This site works only in browsers with SNI support.
Cipher Suites - we are supporting some weak ones

[tmds]

.NET (especially NuGet) has had run-ins with ca-certificates packages on some Linux distros, maybe now it's the website's turn.

wget doesn't like the www.nuget.org cert either:

$ wget https://www.nuget.org/api/v2/package/Microsoft.CSharp/4.7.0
--2021-10-26 16:45:26--  https://www.nuget.org/api/v2/package/Microsoft.CSharp/4.7.0
Resolving www.nuget.org (www.nuget.org)... 52.240.159.111
Connecting to www.nuget.org (www.nuget.org)|52.240.159.111|:443... connected.
ERROR: The certificate of ‘www.nuget.org’ is not trusted.

[simo5]

On F34 wget is linked to GnuTLS, using GnuTLS tools I get this:

$ gnutls-cli dotnet.microsoft.com:443
Processed 150 CA certificate(s).
Resolving 'dotnet.microsoft.com:443'...
Connecting to '13.107.213.40:443'...
- Certificate type: X.509
- Got a certificate list of 2 certificates.
- Certificate[0] info:
 - subject `CN=dotnet.microsoft.com,O=Microsoft Corporation,L=Redmond,ST=WA,C=US', issuer `CN=Microsoft Azure TLS Issuing CA 06,O=Microsoft Corporation,C=US', serial 0x33001c4a935763a41378c71fc60000001c4a93, RSA key 2048 bits, signed using RSA-SHA384, activated `2021-10-07 17:36:19 UTC', expires `2022-10-02 17:36:19 UTC', pin-sha256="TeDSIQAhYgUc8daVbnJwao/TlpHtDsp0AXbJm4/Q0lE="
	Public Key ID:
		sha1:d332371e0f12e662d09ba1303ca9be0e6458c63d
		sha256:4de0d221002162051cf1d6956e72706a8fd39691ed0eca740176c99b8fd0d251
	Public Key PIN:
		pin-sha256:TeDSIQAhYgUc8daVbnJwao/TlpHtDsp0AXbJm4/Q0lE=

- Certificate[1] info:
 - subject `CN=Microsoft Azure TLS Issuing CA 06,O=Microsoft Corporation,C=US', issuer `CN=DigiCert Global Root G2,OU=www.digicert.com,O=DigiCert Inc,C=US', serial 0x02e79171fb8021e93fe2d983834c50c0, RSA key 4096 bits, signed using RSA-SHA384, activated `2020-07-29 12:30:00 UTC', expires `2024-06-27 23:59:59 UTC', pin-sha256="Wl8MFY+9zijGG8QgEHCAK5fhA+ydPZxaLQOFdiEPz3U="
- Status: The certificate is NOT trusted. The received OCSP status response is invalid. 
*** PKI verification of server certificate failed...
*** Fatal error: Error in the certificate.

Seem like OCSP issues of some kind. Unfortunately the --no-ocsp options seem not work, so can't tell if this is a real OCSP issue or something else.

[dagood]

Interesting! Following along with https://security.stackexchange.com/a/73200, I get an interesting result:

$ openssl s_client -connect dotnet.microsoft.com:443 2>&1 < /dev/null | sed -n '/-----BEGIN/,/-----END/p' > acm.pem
$ openssl s_client -connect dotnet.microsoft.com:443 -showcerts </dev/null 2>/dev/null > chain.pembundle
$ openssl x509 -noout -ocsp_uri -in acm.pem
http://oneocsp.microsoft.com/ocsp
$ openssl ocsp -issuer chain.pembundle -cert acm.pem -text -url $(openssl x509 -noout -ocsp_uri -in acm.pem)
OCSP Request Data:
    Version: 1 (0x0)
    Requestor List:
        Certificate ID:
          Hash Algorithm: sha1
          Issuer Name Hash: 0DD55A225F8D2354E325761EEE0D3862F43A9E7B
          Issuer Key Hash: 20E6BE9DCAAB0F68AC9CF0EC7250937411C16EE4
          Serial Number: 33001C4A935763A41378C71FC60000001C4A93
    Request Extensions:
        OCSP Nonce:
            0410FEBF5175CE1BA4BE2FCB6F4058D5218C
Responder Error: unauthorized (6)

Is http://oneocsp.microsoft.com/ocsp misbehaving? Responder Error: unauthorized (6) doesn't seem normal.

[simo5]

Running gnutls-cli with -d 9999 spits these lines at some point (trimmed of other debug statements):

|<2>| looking for key purpose '1.3.6.1.5.5.7.3.1', but have '1.3.6.1.5.5.7.3.4'
|<2>| looking for key purpose '1.3.6.1.5.5.7.3.1', but have '1.3.6.1.5.5.7.3.2'
- Status: The certificate is NOT trusted. The received OCSP status response is invalid. 

[simo5]

Looking at https://oneocsp.microsoft.com/ocsp with a browser I see this:

Our services aren't available right now

We're working to restore all services as soon as possible. Please check back soon.
021N4YQAAAACE429TttJoSYlLJaKGcRPHTllDRURHRTA5MDgARWRnZQ==

[simo5]

And at http:// I get a 500 error, but that may be ok given I am making no real query

[dagood]

I didn't think to check https, nice. I've found some internal team/bug tracking info, but no obvious outage being tracked. I'll send what I found to the Microsoft folks on this thread for followup.

[ueno]

According to the analysis by @ansasaki, the OCSP response is signed with RSA-SHA1, which is disabled by the DEFAULT crypto-policies setting in F34:
https://bugzilla.redhat.com/show_bug.cgi?id=2003363#c4

[tmds]

https://bugzilla.redhat.com/show_bug.cgi?id=2003363#c4

A comment on the ticket says that OpenSSL accepting the certificate may be a bug.

If OpenSSL would not accept it, the impact would be far greater than just affecting wget.
.NET would no longer trust these certificates either.

[dagood]

Thanks all for the detailed investigative help. 🙂 The team is now looking into what it'll take to get an F33/F34 gnutls-acceptable cert set up for the .NET site.

[ChrisSfanos]

Slight clarification - the cert used on the dot.net site is correct. The issue is that the OCSP is returning a SHA1 encoded response that F34 (by default) won't accept (thanks @ansasaki) - we've forwarded the issue to some internal teams who will take a look

[rzio]

According to RFC 6960 , (specifically section 4.3) OCSP responders can return responses signed using RSA with SHA1. RFC 5019 which standardizes "The Lightweight Online Certificate Status Protocol (OCSP) Profile for High-Volume Environments" states that only SHA1 should be supported.
Let's Enrypt had a similar issue filed and reading through it's resolution it seems to me that gnu_tls should support SHA1 for OCSP response.

OneOCSP is compliant to the RFCs similarly to the LE OCSP responder.

[tmds]

@rzio besides saying that gnu_tls should support SHA1, is your reply also meant to imply that you don't plan to change the algorithms used by OneOCSP?

I'm not an expert. Skimming through the RFCs, I get the impression the recommended algorithm changed over time from SHA-1 to SHA-256.

I haven't seen issues besides with the Microsoft sites, or find reports about it. That means other sites are using an OCSP server that supports more than SHA-1.

[tmds]

@rzio I've added your comment to the gnu_tls bugzilla ticket: https://bugzilla.redhat.com/show_bug.cgi?id=2003363.

[ueno]

RFC 5019 which standardizes "The Lightweight Online Certificate Status Protocol (OCSP) Profile for High-Volume Environments" states that only SHA1 should be supported.

I think this is not relevant; the RFC only states:

Clients MUST use SHA1 as the hashing algorithm for the CertID.issuerNameHash and the CertID.issuerKeyHash values.

and nothing else, while we are talking about SHA1 usage in signatures. Also note that it is a distro-wide decision to disable SHA1 signatures by default, not a decision by the single TLS library; the detailed rationale can be found at the change proposal.

[tmds]

:/

[ueno]

That is probably https://bugzilla.mozilla.org/show_bug.cgi?id=1745600 (and https://bugzilla.mozilla.org/show_bug.cgi?id=966856#c8).

[RokeJulianLockhart]

Is addition of '--no-check-certificate' to the script too insecure to be feasible?

[RokeJulianLockhart]

This prevents compilation of PowerShell.

[tmds]

Is addition of '--no-check-certificate' to the script too insecure to be feasible?

Yes, this is a no-go. When you add this, you can't consider anything that happens next to be secure/trusted.

[rzio]

Update from the OneOCSP side: Due to a new CAB forum requirement, OneOCSP will switch the algorithm to SHA-256 by 2022-5-31.

[tmds]

Closing as the issue is fixed.