Feature Request: Mutual TLS with ACM PCA-managed Certificates

[bigdefect]

If you want to see App Mesh implement this idea, please upvote with a 👍.

Tell us about your request
Today App Mesh has support for retrieving server/upstream certificates and trust bundles for downstreams/clients from ACM Private Certificate Authority. The initial release of Mutual TLS support (#34) will enable retrieving TLS materials from the file system or an SDS provider.

We can provide parity for supporting ACM PCA for client certificates and server trust bundles.

Which integration(s) is this request for?
All

Are you currently working around this issue?
Certificates and trust bundles from ACM PCA could be exported to the file system. SPIRE supports using an ACM PCA as an upstream root to sign its subordinate CA certificates.

[sermilrod]

This is our main showstopper when trying to adopt App Mesh, since all our services communicate using mTLS. It has been opened for a long time. Any updates or new developments in this front?

[herrhound]

@sermilrod -- today, you can use certificates from ACM PCA to establish mutual TLS with App Mesh-managed services. However, they have to be distributed to Envoys manually, via file system, or via Envoy's SDS API. For customers using K8s, we have an integration with SPIRE: https://aws.amazon.com/blogs/containers/using-mtls-with-spiffe-spire-in-app-mesh-on-eks/

We are planning to build a better integration with ACM PCA for mTLS, similar to what we have for TLS use cases. However, there are some blockers that are being actively worked on by the ACM PCA team. I don't have an exact timeline, unfortunately.

What is the orchestrator that you're using, and how often do you rotate certificates for your mTLS workloads?

[sermilrod]

Hi, Thanks for the quick response! I am using ECS fargate with app Mesh. Injecting by myself the PCA certificates will force me to develop a way to rotate the certificates when they expire. How do I make envoy pick that up? Do you have a recommended way of doing it before app Mesh new features are out? In addition there is another problem, exporting the private certificate from ACM and injecting it into the envoy container does not work out of the box. The private key is encrypted as it is mandatory to pass a passphrase in order to export the certificate with the key and its chain. How does envoy consume this passphrase? I have not been able to find this in the documentation. If I remove the passphrase this is what I get: Failed to load private key from /keys/service-key.pem, Cause: error:0b000074:X.509 certificate routines:OPENSSL_internal:KEY_VALUES_MISMATCH

[herrhound]

Hi Sergio @sermilrod,

We are working on a reference architecture for enabling mTLS with App Mesh on ECS / Fargate, targeting mid-June. In essence, we'll provide a custom CFN resource that mounts EFS, implements the CA and creates signed certs for the services in the EFS, and then the services will mount different access points to get their own certs mounted to the Envoy proxy container. It will include the automatic certificate renewals as well.

Speaking of the use case that you've outlined with a certificate exported from ACM, I believe it won't work. ACM does not allow you to export the key material along with the cert so you can't make it work with Envoy. You can export a certificate from ACM Private Certificate Authority for that use case.

[sermilrod]

Hi @herrhound thanks for timelines, I really appreciate it.

I just found it's been a long time since envoy supports encrypted private keys with passphrases https://github.com/envoyproxy/envoy/pull/5175/files but the virtual node API does not allow you to pass that in. It only exposes certificate_chain and private_key. Is there a way to customise the envoy config to pass that config in the short term?

Thanks a lot in advance

[ulziibay]

@sermilrod were you able to find a workaround to this issue of using passphrase-protected private keys in the envoy proxy?

I imagine I can generate the certificate materials using openssl like in the walkthrough and import it to ACM as well as put it into the proxy container via local file system, when I cannot directly use the certs exported from ACM for the proxy mTLS. Is that recommended @herrhound ?

[sermilrod]

@sermilrod were you able to find a workaround to this issue of using passphrase-protected private keys in the envoy proxy?

I imagine I can generate the certificate materials using openssl like in the walkthrough and import it to ACM as well as put it into the proxy container via local file system, when I cannot directly use the certs exported from ACM for the proxy mTLS. Is that recommended @herrhound ?

We ended up writing a lambda to react to ECS events and issue certificates on-demand. In addition we had to add an init container to feed envoy with the certificates created before it can start.

This is effectively a serverless Private CA solution and it is far from ideal. We do expect AWS to provide an out-of-the-box solution as this makes App Mesh hard to adopt. @herrhound any updates from this feature ? Could you provide more accurate timelines?

[RafalMaleska]

We are using ACM for TLS with AppMesh but our security guidelines require mTLS.
A workaround with additional init containers/lambda and ecs events is not possible in our case.

Will the Issue be listed on the RoadMap?

[RafalMaleska]

Found this: https://aws.amazon.com/blogs/security/how-to-use-acm-private-ca-for-enabling-mtls-in-aws-app-mesh/

This allows the usage of mTLS in K8S but it's making the usage of AppMesh far more complex. Therefore it would benefit to push this issue onto the Roadmap.

[jlambert121]

As a customer and someone wanting this feature, is there feedback I can provide to see about getting this on the roadmap?

[bigdefect]

@jlambert121 I'm no longer on App Mesh, but I can try to answer. If you're part of a business and have a technical account manager with AWS, that's a great way to funnel input to the service teams. That being said, the activity here is a good signal, even just the +1's to the top level comment help rank feature interest.