Present Consul server cert chain when using Vault secrets backend #1251
Conversation
Previously, the helm chart only rendered the server leaf cert when using the vault PKI backend. If your PKI was set up to use one or more intermediate certs, this meant that Consul wasn't presenting the full intermediate CA chain. This change includes all intermediate CA certs in alongside the leaf cert. It skips the root cert, since that's presumably already going to be in your system's trust store. If there's a better way to do this, I'm all ears! This is the first time I've dealt with helm and vault templating.
This doesn't actually test that the template is doing the _right_ thing, but at least we can verify that the rendered template makes some sense.
|
|
There was a problem hiding this comment.
@brucec5 Thank you so much for raising this issue, giving us detailed information, and submitting this solution. I have validated through a manual script i created in this repo: https://github.com/jmurret/consul-vault-offline-ca-root.
As mentioned in that README, the browser on a mac gives some fits related to the leaf cert not being standards compliant, but the trust chain is properly displayed in the browser and it passes openssl verification. I think the mac browser issues are more related to the strictness of this development, self-signed cert, so I'm not worried about it.
Also, I copied these changes and ran them through our acceptance tests and nothing failed.
I think for the moment manual verification plus the bats unit tests are sufficient. I'll talk to the team if we want to create an acceptance for it, but since this is an area of code that won't change much and would not really be affected too much other Consul configurations or any timing conditions or anything similar that we see in Kubernetes environments, we may just leave the BATS tests as sufficient, but we'll see.
Once again, thank you so much. I'll get another team to review. Cheers. -John
|
@brucec5 can you please sign the CLA? It's the last hurdle to getting this merged and ready for release. 🙏 |
|
Sorry for the delay on getting back to this one! I got distracted by some other stuff and kinda forgot about this PR. The CLA is signed now. |
Changes proposed in this PR:
I was told by some HC support/CSM folks to open this as a PR. I don't know if this is the best way to accomplish this, or if the testing is sufficient, but hopefully this is useful. This was initially just meant to be a POC to show the engineering team what I did to address the issue on my end.
Currently, only the leaf cert gets presented by the Consul server (both UI and API). This works fine if you have a self-signed cert, but if you're using Vault as a secrets backend with an intermediate CA (and perhaps an offline root CA), then you can't currently verify the full chain.
The fix is to loop over the CA chain and append the intermediate cert(s) to the end of the server cert. This lets Consul present the full chain, which allows for proper validation.
This is related to #879, but only addresses the Vault secrets backend. Different work will need to be done to support the Kubernetes secrets backend.
How I've tested this PR:
I set up an offline root CA cert and used that to sign an L1 ICA in Vault, and used the L1 ICA to sign an L2 ICA. I then used that L2 ICA to issue certs for Consul. I added the root CA cert to my system's trust store and was able to access the UI in the browser without running into a TLS verification error.
It's been a while, but I'm pretty sure I also tested setting up a root CA directly in Vault and using that to issue certs for Consul.
Please feel free to suggest additional testing.
How I expect reviewers to test this PR:
openssl s_client -connect localhost:8501 -CAfile path/to/root-ca.pem, which should return successfullyChecklist: