Certificate rotation

[danefalvo]

Prerequisites

• I have searched open and closed issues to make sure it isn't already requested
• I have discussed this in #feature-discussion or #backlog and the consensus is that this is something we plan on adding in the near future
• I have written a descriptive issue title
• I have linked the original source of this feature request
• I have tagged the issue appropriately (area/*, kind/enhancement)

The enhancement

What is the problem this solves or benefit it gives

There is multiple scenarios where this issue will become apparent however possibly the most common scenario will be the following:

If a customer has a bunch of polling tentacles connected using websockets, a Tentacles "Trusted Octopus Server thumbprint" value is not set to the Octopus Server thumbprint. It has to be set to the certificate fingerprint as described here: https://octopus.com/docs/infrastructure/deployment-targets/windows-targets/polling-tentacles-web-sockets#thumbprints

This will mean that the SSL certificate will likely expire every 2 years (possibly sooner) and the trusted fingerprint will need to be changed on all tentacles. This also means that the change to the new trusted certificate needs to happen at roughly the same time as the certifcate change on the server.

It is possible to automate the trusted certifcate task by using the Update-Trust Command. ( https://octopus.com/docs/octopus-rest-api/tentacle.exe-command-line/update-trust ). This causes multiple issues:

1. Any offline tentacle will still only trust the old Certificate. When they come back online they will not update to trust the new certifcate, as the new certificate will already be in place and any tentacles attempting to connect using the old certifcate fingerprint, will fail.
2. It also means that the entire change has to be done pretty much at the same time.

The other option is to add a second trusted Octopus Server so that at any point in time, the tentacle will have valid certifcate that it trusts.
Unfortunately the configuration can't be completed, using the Tentacle CLI.

The following command will work:

Tentacle.exe configure --trust=xxxxxxxx

however this will result in the a configuration file similar to this one:

As you can see, without

• the CommunicationStyle set to 2
• the address set to an actual address

The second trusted Octopus Server will not work.

Tentacle.exe configure --trust=xxxxxxxx does not accept any of the following parameters:

--address
--comms-style
--server

Proposed solution

Make it mandatory for Tentacle.exe configure --trust=xxxxxxxx to include the following additional parameters:
--address (or --server)
--comms-style

Workarounds

Currently the only solution to this issue is to set up a scheduled task that will:

• Update-trust
• Restart Tentacle

This scheduled task needs to be set with two triggers.

1. At the exact time the certifcate gets replaced (for all online tentacles)
2. At next logon (for all offline tentacles)

Links

https://octopus.zendesk.com/agent/tickets/62880
https://octopus.zendesk.com/agent/tickets/2403
https://octopus.zendesk.com/agent/tickets/19737
https://help.octopus.com/t/octopus-server-trust-certificate-rotation/22604/15
https://octopus.zendesk.com/agent/tickets/64930

[tothegills]

This issue revolves around the experience our customers our having when attempting to rotate their Octopus Server certificate. Especially a problem when using web sockets.

I think the solution proposed in this issue has some holes. Adding new mandatory parametes for the configure --trust command will be a breaking change with huge implications. The steps to perform a certificate rotation will still be cumbersome: update the trust on all Tentacles (hopefully we don't miss any), restart all Tentacles, rotate the certificate, remove the old certificate from Tentacles, restart all Tentacles. It's not pretty.

I propose we make certificate rotation a small deal. A small deal might look like a UI where the new certificate and rollover date can be configured. Tentacles automatically begin trusting the new certificate and revoke the old certificate when they recognise the new certificate is being presented. The Octopus Server replaces its certificate when the rollover date is reached.

[tothegills]

Customer feature request: https://octopusdeploy.slack.com/archives/C033W4273/p1645101524789899

[paraicoceallaigh]

Customer issue (polling tentacles specifically): https://help.octopus.com/t/octopus-server-exe-export-certificate/28745/23