When deploying to IIS with a HTTPS binding and updating a certificate, an unrelated inaccessible certificate store can cause deployment to fail

[patrick-smergut-octopus]

Team

• I've assigned a team label to this issue

Severity

Currently one report of issue, will affect anyone with an AD certficate store

Version

2022.1.1630 (cloud)

Latest Version

No response

What happened?

When replacing/updating a certificate for HTTPS bindings when deploying an IIS website via the Deploy to IIS step, an inaccessible certificate store (e.g. UserDS / Active Directory User Object in the certmgr.msc) may cause the deployment to fail, even if the certificate exists in a different, accessible certificate store.

A failure to open a certificate store should be gracefully handled, moving onto other certificate stores, so that a store that isn't relevant to the deployment doesn't fail the deployment entirely.

Reported here [Internal]: https://octopus.zendesk.com/agent/tickets/83394

Reproduction

1. Restrict permission to a certificate store (e.g. UserDS)
2. Set up a process to deploy to an IIS website using a HTTPS binding and a certificate that is installed in some other, accessible certificate store
3. Run the deployment
4. Update the bindings to use a new certificate in the deploy to IIS website step and make sure 'Replace existing bindings' is selected
5. Create a new release and run the deployment

Error and Stacktrace

17:09:26   Verbose  |       Executing feature-class 'Calamari.Deployment.Features.IisWebSiteBeforeDeployFeature'
17:09:26   Error    |       System.Security.Cryptography.CryptographicException: The system cannot find the file specified.
17:09:26   Error    |       at System.Security.Cryptography.X509Certificates.X509Store.Open(OpenFlags flags)
17:09:26   Error    |       at Calamari.Deployment.Features.IisWebSiteBeforeDeployFeature.FindCertificateInLocalMachineStore(String thumbprint)
17:09:26   Error    |       at Calamari.Deployment.Features.IisWebSiteBeforeDeployFeature.EnsureCertificateInStore(IVariables variables, String certificateVariable)
17:09:26   Error    |       at Calamari.Deployment.Features.IisWebSiteBeforeDeployFeature.EnsureCertificatesUsedInBindingsAreInStore(IVariables variables)
17:09:26   Error    |       at Calamari.Deployment.Features.IisWebSiteBeforeDeployFeature.Execute(RunningDeployment deployment)
17:09:26   Error    |       at Calamari.Deployment.Conventions.FeatureConventionBase.ExecuteFeatureClasses(RunningDeployment deployment, String feature)
17:09:26   Error    |       at Calamari.Deployment.Conventions.FeatureConventionBase.Run(RunningDeployment deployment)
17:09:26   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
17:09:26   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
17:09:26   Error    |       Running rollback conventions...

More Information

This is likely related to a similar issue we've already fixed #6378

• The 6378 issue was to do with a problem with getting Octopus to correctly configure the binding of an existing certificate to the IIS Site
• This was due to a specific configuration of cert store, which integrates with Active Directory, causing the cert locator code to bomb out.
• We fixed the 6378 issue in the spot where this happens in the post-deploy Powershell code of the IIS step
• This new issue deals with attempting to use Octopus to update the certificate (and of course, the IIS bindings, if the thumbprint has changed) in the stores
• This happens in a different part of the code, in Calamari/C#, which happens pre-deploy

Workaround

We haven't tested this workaround yet, but it feels like it could work:

• Add an Import Certificate step to run before the Deploy to IIS step, to deploy the certificate to the target store location.

As this goes straight to a specific store rather than trying to iterate over each of the stores on the machine (which is what causes the problem, combined with the UserDS store type), this might help work around the problem until we can fix the root cause.