Consul ESM cannot validate Auto Encrypt Agent Certificates with expired cross signed certificates

[lawliet89]

I am running Consul on Kubernetes using the official Helm chart. I have Auto Encrypt turned on and I recently rotated Connect CA to use Vault.

This has resulted in certificates issued to Consul agents to contain certificate that was cross signed by the old Connect CA. The cross signed certificate has since expired and Consul ESM emits errors like

2020/09/03 10:08:14 [ERR] error getting leader status: "Get https://10.1.1.40:8501/v1/status/leader: x509: certificate has expired or is not yet valid", retrying in 10s...

I had to set CONSUL_HTTP_SSL_VERIFY=false for Consul ESM to work.

This does not seem to be a problem for Consul Template 0.25.1. I noticed that Consul Template depends on Consul API v1.4.0 and SDK v0.4.0 whereas Consul ESM depends on Consul API v1.2.0 and SDK v0.4.0. I couldn't really identify the changes between 1.4 and 1.2 that might have fixed this. Could a bump to at least API 1.4 fix this?

[lornasong]

@lawliet89 - thanks so much for writing this issue. Yes, I think it would be a good idea to upgrade the Consul API version. Let me take some time to research and understand any impact from upgrading and which version to upgrade.

I did a little searching to see if I could find changes between 1.2 and 1.4 that could potentially fix the issue you describe. One potential is: Add option to set TLS options in-memory for API client. Looks like it actually links to a Vault issue you created :). When you have a chance, would you mind taking a look and letting me know what you think?

Thank you!

[lawliet89]

I'm not sure.

In any case I tried updating the dependencies myself but it involved a fair bit of changes for logging because consul changed its logging package and so I didn't proceed further.

[lornasong]

Thanks for trying out updating the dependencies, @lawliet89. That's helpful to know about the logging changes and good to keep in mind when looking into the upgrade. We have an open issue to update logging #82, which might be related. Please feel free to comment if you have any additional details. Thanks!

[eikenb]

Hey @lawliet89, sorry for the long delay getting back to this. We all got pulled off on a different project for a while and I'm only now getting some time for ESM.

I'm going to look into updating these dependencies, with whatever work that entails (eg. mentioned logging module is the first up).

[lawliet89]

Thanks. I no longer have access to the cluster that experienced this so I don't think I'll be able to reproduce or verify any fixes. I think a dependency update should be all that's necessary.

[eikenb]

Hey @lawliet89, thanks for the feedback. Good to know you concur that a dependency update fixes it. Hope you've moved on to bigger, better things. Thanks again!