Powershell script failing with Cannot create TLS/SSL secure channel - managed to restart Cloud Service roles

[cadumi]

Steps to reproduce

Behavior reproducible under WS 2016 and 2019 with SEPTEMBER ROLLUP installed.
Issue not reproducible without rollup.

Customer is using a Virtual Machine to run the DevOps pipeline, where he is running a powershell script to manage the classic cloud service, he is using the below script: https://gallery.technet.microsoft.com/scriptcenter/Reboot-Cloud-Service-PaaS-b337a06d

The error customer is seeing when attempting to call Azure management endpoint (management.core.windows.net) is a TLS/SSL verification failure due to untrusted root in the certificate chain. "System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel."

We got repro in lab even with one cmdlet Import-AzurePublishSettingsFile .\1.publishsettings
It created certificate in user storage and removed private key immediately.
I collected IDNA trace and found that during the running this script powershell.exe calls ncrypt!NCryptImportKey and then after some time clr!X509Helper::DeleteKeyContainer

As we did not change anything in this script in September, but customer opened ticket on 8th of September, that means something was changed on client side.

I tried to get repro in lab on W10 1809 with August updates. And everything worked fine with the same script with the same .publishsettings file

As soon as i installed 9B cumulative update for .NET, i got repro with exception.

It looks like in .NET 9B update we have a BUG, that causes non-correct work with certificates.

For W10 1809 /WS 2019 - this is https://support.microsoft.com/en-us/help/4576627

For W10 1607 /WS2016 - this is https://support.microsoft.com/en-us/help/4576479

in our public article regarding the .NET update https://support.microsoft.com/en-us/help/4576479/kb4576479-cumulative-update-for-net-framework

CLR1 | - Addresses an issue in some 32 bit apps where, in certain scenarios, the JIT might omit a function call - Improved support for cleaning up private temporary certificate keys

i discussed the issue with Developer(Jeremy Barton), who made fix in September .NET update.

His conclusion:

"If the script is loading a certificate from the certificate store and it starts failing after some unpredictable amount of time, it's that whatever loaded the certificate into the certificate store didn't use PersistKeySet and it finally got garbage collected, erasing the "temporary" private key file".

This behavior is described here:

https://docs.microsoft.com/en-us/troubleshoot/dotnet/framework/install-pfx-file-using-x509certificate

Cause

When the certificate is installed by using the X509Certificate or X509Certificate2 class, X509Certificate or X509Certificate2 by default creates a temporary container to import the private key. The private key is deleted when there's no longer a reference to the private key.

So guess Azure PS PGs should double check the code of all scripts, which use certificate importing functions regarding the PersistKeySet

Internal escalation: 206414717

-- | --

## Expected behavior

```none
Script from Devops pipeline should run with update installed as before.

Actual behavior

The error customer is seeing when attempting to call Azure management endpoint (management.core.windows.net) is a TLS/SSL verification failure due to untrusted root in the certificate chain. "System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel."

Environment data

Version: 5.1.14409.1005

[vexx32]

It sounds like this is:

1. Primarily an issue with Windows PowerShell, which is no longer being developed (does this same issue happen in PowerShell 7? Does the behaviour still vary across Windows patch versions with PowerShell 7?)
2. Not really an issue in PowerShell itself, merely the result of a change in Windows or in the module Windows provides to interact with certificates.

Is that accurate? If so, I'm not sure there's much the powershell team can do here. 😕

[rjmholt]

Hi @cadumi, it looks like the issue you're experiencing is one with Windows PowerShell, rather than PowerShell 7+, which is what's tracked here.

Feel free to reach out to me over Teams or email instead so we can redirect you as needed.

[riccom76]

@rjmholt , here's the issue is more with the Azure PowerShell cmdlets which are failing since the cert chain cannot be found due to the above error.
It looks like the cert chain, used by the Azure PS commands, was stored without the "PersistKeySet".

The question is: is this cert chain, used by Azure PS, 1) stored when the Azure PS is installed or when AZ login run? or 2) is this cert chain OS native?
Whether the first is true, we need to store that cert chain using "PersistKeySet".

[vexx32]

I think that would need to be handled inside the Azure module iself? I don't think the powershell team have any control over that.

Might be better to ask in the Azure/azure-powershell repo?