Replacing expired trust anchor fails

[D3N14L]

Bug Report

What is the issue?

The trust anchor certificate expired and I tried to replace it following the procedure in: https://linkerd.io/2/tasks/replacing_expired_certificates/ but the command in the documentation failed, due to the certificate expiration it is trying to fix.

How can it be reproduced?

Setup a new linkerd instance with a certificate that expires very soon.

Logs, error output, etc

I created a complete set of new certs with step. Including issuer cert and key. and tried to update my manifests by using the upgrade command with --force option per documentation.

linkerd upgrade \
    --identity-issuer-certificate-file=./issuer.crt \
    --identity-issuer-key-file=./issuer.key \
    --identity-trust-anchors-file=./ca.crt \
    --force > linkerd.yaml
× Failed to build upgrade configuration: could not fetch existing proxy injector secret: x509: certificate has expired or is not yet valid: current time 2020-07-28T11:15:03+02:00 is after 2020-07-28T07:01:20Z - Current Time : 2020-07-28 11:15:03.789613 +0200 CEST m=+0.274195056 - Invalid before 2019-07-29 07:01:00 +0000 UTC - Invalid After 2020-07-28 07:01:20 +0000 UTC
For troubleshooting help, visit: https://linkerd.io/upgrade/#troubleshooting

It failed because... the certificate that I wanted to replace is expired, well...

linkerd check output

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
× trust anchors are within their validity period
    Invalid anchors:
	* 1 identity.linkerd.cluster.local not valid anymore. Expired on 2020-07-28T07:01:20Z
    see https://linkerd.io/checks/#l5d-identity-trustAnchors-are-time-valid for hints

Status check results are ×

Environment

• Kubernetes Version:v1.15.11
• kops 1.16.1
• Host OS: ubuntu
• Linkerd version: 2.7.1

Possible solution

Deleted and reinstalled linkerd with the new certificates.

Additional context

[adleong]

Thanks for this report, @D3N14L. You're right, --force should definitely allow the replacement of certificates which are expired. I will look into this.

[adleong]

After some investigation, it looks like the issue here is actually that your proxy-injector TLS certificate is expired as well and this is preventing you from being able to replace your issuer credentials. I have filed #4813 which should prevent this from happening in the future.

In the meantime, you can manually delete the linkerd-tap-tls secret from the Linkerd namespace and then you should be able to successfully run the linkerd upgrade command to replace your issuer certificate. (you may need to delete the linkerd-sp-validator-tls secret as well if it is also expired)

kubectl -n linkerd delete secret/linkerd-proxy-injector-tls

Let me know if you run into any difficulties with this!

[ihcsim]

@D3N14L Have you had a chance to try out @adleong suggestion?

[D3N14L]

@adleong @ihcsim Sorry for the late response - I did not had the chance to test @adleong s suggestion. Our prod cert expired closely after the dev, so I was kind of in a hurry. The only solution that I could think of was reinstalling linkerd and restarting all the pods. That, although it produced a short downtime, worked as expected.

[ihcsim]

@D3N14L Thanks for the updates. In that case, can we close this issue? We have a follow-up issue at #4813, to handle the webhooks' CA bundle.

[D3N14L]

Fine with me - thanks for looking into it!