ACM service controller

[mdykes-gw]

New ACK Service Controller

Support for ACM

List of API resources

List the API resources in order of importance to you:

1. Certificate

[jaypipes]

@mdykes-gw can you elaborate on how you'd envision an ACK service controller for ACM working? I suppose the only resource in the ACM API is the Certificate resource. However, instead of a Create call, there is only an ImportCertificate API call. There are API calls like ExportCertificate that don't make sense in a Kubernetes resource model world (there is no state to reconcile...).

Were you thinking of implementing your own kind of certificate renewal using an ACK service controller for ACM and some other script or Kubernetes operator?

[Vrtak-CZ]

From my point of view it can initiate creation of Certificate so this certificate can be used with AWS LoadBalancer Controller. The problem is that this can be tricky because it will probably also need access to Route 53 resource for validation and it will work only for domains hosted in Route 53.

[Comradin]

@Vrtak-CZ But isn't this basically the exact use-case for many EKS users?

We have EKS, Route53, and the Loadbalancer Controller and no automatic certificate issuer for dynamic scopes. We had to fall back to pre-defined certificates using tools like pulumi or terraform. This breaks the envisioned workflow of just creating a development instance with a valid certificate chain

Route53 in our setups have an authoritative domain like dev.example.com

The developers should be enabled to deploy an ingress host for my-service.my-dev-namespace.dev.example.com and everything works in an automated way.

[ack-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/aws-controllers-k8s/community.
/lifecycle stale

[Vrtak-CZ]

/remove-lifecycle stale

[Pitta]

I came here looking for exactly this workflow. Would prefer to not use a wildcard cert. Use wise, being able to include an annotation to create the cert in acm would be useful.

metadata:
  annotations:
    aws.acm.kubernetes.io/create: true

The aws-load-balancer-controller should wait till there is a valid cert if using the certificate lookup method. So if this listened for ingress objects and acted on that annotation, it would jive with the workflow of the aws-load-balancer-controller.

---

Or just use a similar Certificate kind used by CertManager and grant the controller to specified Route53 hosted zones and Cert Manager for creating new certs.

[RedbackThomson]

@Pitta
I think these are good suggestions, I'm just not sure if ACK is the place for them.

AWS already supports using ACM for private certificates through the cert-manager issuer - https://github.com/cert-manager/aws-privateca-issuer/

Perhaps some integration between that issuer and the aws-load-balancer-controller would be more appropriate? It is not within the scope of ACK to interact with Ingress resources - only our own custom resources.

[sbkg0002]

Why the forced push on the private ca?

If cloudformation or the cli can do it, this should also be able right?

[Pitta]

I've been working on a simple controller to handle this based on kopf at work. Hoping I can convince the org to start making some stuff open source, or at least let me fork the code properly so I can make it public outside the org.

That said, the logic is pretty simple. I haven't done the service account yet, but the operator worked in my local testing.

[sbkg0002]

Thanks for sharing @Pitta !looking forward.

[RedbackThomson]

@sbkg0002 Yes CloudFormation and the CLI are both able to create certs, and an ACM ACK controller would be able too as well. However, we don't have plans to extend ACK out any further than the control plane of ACM, for example attaching certificates within a Kubernetes environment.

I think there are other good tools (like cert-manager) that handle the K8s part of it once the certificate has been created by an ACK custom resource.

[Pitta]

If cert-manager did what I was asking for in a way that was clear that others have done, I'm all for it.
Nothing I've found in my travels suggests that anything out of the box will do ACM public cert management.

[vijtrip2]

/lifecycle frozen

[olemarkus]

The ACM + route53 validation + Load balancer controller use case is definitely very interesting to us. Not having this functionality prohibits us from using ACM and ALBs.

[ohookins]

I also have this use case and the most recently linked issue is mine. Like another previous commenter I'm a bit mystified why there seems to be more support in the ecosystem for ACMPCA rather than public ACM certificates - so many use cases for dynamic environments require a load balancer, publicly-verifiable certificates and matching DNS.

To be honest I don't know why the ALB, ACM and Route53 services aren't more tightly integrated to make this a one-click operation, but that's a separate feature request ;)

[erhudy]

I too am interested in this. I think it's worth distinguishing between the IssueCertificate and RequestCertificate API calls, because they do different things. The cert-manager plugin for ACMPCA seems to only do IssueCertificate, which is an ACMPCA-exclusive call - this requires you to generate a key and CSR and submit it to the PCA, and then you get a certificate back. It does not seem to support RequestCertificate, which can be used both for ACM public certs and also for getting certs from ACMPCA that you want to use in other managed services like ALB.

This RequestCertificate gap is what I would like to see solved; we use ACMPCA now but have to tell people that they need to provision certs themselves through Terraform/API/console and get the ARN, then use that ARN in the annotation for the LB controller. It's not wretched, but the ideal workflow would be annotating a Service or Ingress in a way that tells the LB controller to deal with requesting the certificate itself and figuring out the SANs based on the NLB hostname annotation or Ingress hosts, respectively.

[matheushent]

If it matters, there still are people interested in this, e.g. me!

[migueleliasweb]

It's 2023 now.... 1 year and 2 months of this thread. This shouldn't be taking this long.

It's quite sad that some are still trying to untangle all the details of an advanced implementation (ACM with Private CAs + LB attachment) but the basic functionality of just creating a simple public ceritificate using DNS challenge is not even supported yet.

Can we just agree of having a simple support for creating public certs first then we can iterate over that to then add support for more advanced cases?

Happy 2023 🎉

[jaypipes]

but the basic functionality of just creating a simple public ceritificate using DNS challenge is not even supported yet.

If only it were actually a simple thing. :)

Can we just agree of having a simple support for creating public certs first then we can iterate over that to then add support for more advanced cases?

After digging into the ACM APIs, I think we could support Certificate resources with the RequestCertificate API as the Create operation, DeleteCertificate as the Delete operation, DescribeCertificate API as the ReadOne operation and UpdateCertificateOptions as the only Update operation.

We would need to add a caveat, though:

If no certificateAuthorityARN is specified, the Certificate will be a public one. In this case, validationMethod will be hard-coded to DNS (instead of Email) and the number of domainValidationOptions will be limited to 5 (because more than 5 mean that email verification is needed, and that's not an automateable thing.

We could handle the ImportCertificate path at a later date.

@migueleliasweb would that meet your minimum use case needs?

[migueleliasweb]

Thanks for jumping in, @jaypipes . I think you've nailed the problem.

I did basically the same digging as you did and I didn't think that was overly complicated. I'm sure there's a lot of people here in the AWS team (and in the broader community) that are on top of the APIs, so my rationale was that the main problem here is having a plan in the first place and not really implementing it. That's basically why I thought it was sad this thread is taking over a year to output any sort of outcome.

Just to be clear, the usecase you mentioned won't fully solve my usecase as I would still like to have some kind of way to attach this to a LB (I was thinking there could be something like a CertificateAttachment kind we could use to attach a given cert to a LB asynchronously).

Have said that, this is lightyears better than nothing! I will take it!

Taking smaller steps like this will give all the involved parties more confidence they're heading the right direction. This (from my point of view) is far better than trying to foresee 2..3..5..10 steps ahead and ended up overcomplicating something that could have given value to the community much sooner.

[tavlima]

That would be awesome, @jaypipes!

FWIW, this is the cert-manager feature we use. Perhaps it would make sense for this service controller to export any certificate generated by a given cluster issuer?

A couple of caveats, though, is reconciling these certificates (as Ingress resources come and go), as well as the case of multiple EKS clusters in the same account. But I guess you already thought of all these. 😅

[calvinbui]

great work @jaypipes

are there plans to automate validation through route 53 or by a helper utility?

[jaypipes]

great work @jaypipes

are there plans to automate validation through route 53 or by a helper utility?

Hi Calvin! At this time, no, I don't have plans to do this. As we complete the route53 controller, however, we can certainly look into this. Would it be possible for you to create a separate GH issue for the route53 validation behaviour feature request, since this GH issue is for the ACM service controller generally?

[akamac]

Our use case is to import certs, issued by cert-manager, to ACM.
It would be great if the operator can do this.

[FernandoMiguel]

Our use case is to import certs, issued by cert-manager, to ACM.
It would be great if the operator can do this.

@akamac ACM certs are free, why not issue new ones instead?

[akamac]

Our use case is to import certs, issued by cert-manager, to ACM.
It would be great if the operator can do this.

@akamac ACM certs are free, why not issue new ones instead?

Because AWS no longer issues certificates for ru zone, while Let's Encrypt does.

[mahadh02]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

[jaypipes]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however.

[mahadh02]

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however

@jaypipes , my apologies for incorrectly quoting.. this feature will be very beneficial and looking forward for its availability.

[kstevensonnv]

@jaypipes Echoing what the initial comments said.

We're moving to EKS and I was looking to use this controller to create a certificate and complete validation.

If this controller can satisfy that, we could use AWS Load Balancer Controller to act on the ingress resource with simply a host set and it would all just work nicely (in theory).

Now I'm looking at implementing what we currently have outside EKS which feels like a step back.

[FuriouZ07]

DNS validation via R53 record set is the last missing piece for creating a service with ELB and certificate with r53 alias through Kubernetes manifests...unfortunately, as long as this is missing, the acm-controller is not usable efficiently.

[Gianluca755]

@FuriouZ07 shouldn't that be done with external-DNS + acm-controller?
General question: does the acm-controller work with annotations on an ingress resource or it needs a crd?

[FuriouZ07]

@FuriouZ07 shouldn't that be done with external-DNS + acm-controller? General question: does the acm-controller work with annotations on an ingress resource or it needs a crd?

Yeah, external-dns could create a R53 record. But it needs to query the cname name&value from the ACM certificate and then create a record with these data. As far as I know, external-dns does not support such a use case to automate the creation&verification completely.

I've used the CRD for creating the certificate and did not try to use annotations.

[indrekj]

I'm also not finding a way to automate the R53 DNS validation. Everything else worked well, but the validation is a blocker for us as well.

[tavlima]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however.

Hey, @jaypipes. Happy new year. 😄 Any updates on this?

[john-r-swyftx]

@jaypipes I can create a validated cert by doing the steps in #1904 (comment)

But there is the missing part of getting the resource record info to create the correct RecordSet resource. Can that info be put onto the Certificate status like aws acm describe-certificate --certificate-arn <arn> does?

I could work with that for now

[a-hilaly]

Can that info be put onto the Certificate status like aws acm describe-certificate --certificate-arn does?

@john-r-swyftx Resource recordsets are now shown in the Certificate status - checkout acm-controller v0.0.12/13 https://github.com/aws-controllers-k8s/acm-controller/pull/29/files

[tavlima]

@tavlima I can work on an ImportCertificate workflow for ACM, sure. Unfortunately I won't be able to get to this for probably another couple weeks, though.

@tavlima , Wondering if you have any updates on the ImportCertificate workflow? TIA

@mahadh02 it was me who was working on this :) And I have not had the time or resources to get to it unfortunately. I may have some time towards the end of May to tackle this, however.

Hey, @jaypipes. Happy new year. 😄 Any updates on this?

Hey, @a-hilaly, do you happen to have any updates on this? Should I create another issue, for better tracking?

[gregsidelinger]

I did some playing around with the controller today and it is looking nice so far. I have a couple of questions/feature requests.

• Can the cert status be shown with a simple kubect get certificates.acm.services.k8s.aws. A simple Ready or Issuing status would be nice to avoid having to look at the full status.
• Should the domainName and subjectAlternativeNames be editable? Right now changing them does nothing to pending or ready certs. I see logs they say it is different but no new cert is created in ACM. Was wondering what the desired behavior is suppose to be if those fields are designed to changeable.

[is-it-ayush]

Our use case is to import certs, issued by cert-manager, to ACM.
It would be great if the operator can do this.

@akamac ACM certs are free, why not issue new ones instead?

Because AWS no longer issues certificates for ru zone, while Let's Encrypt does.

Been stuck with exactly this! My setup is non-EKS kubeadm cluster running on EC2 with aws-load-balancer-controller, cert-manager and LetsEncrypt CA. I hope the ACM controller or Load Balancer controller supports,

• Automatically uploading and deleting LetsEncrypt issued certificates to the ACM such that ALB can auto-discover them based on host name in TLS spec.
• Can annotate the certificate CRN in NLB such that NLB can work with TLS.

Without either, TLS on Load Balancer's is pretty much impossible to automate on non-EKS (tested) or even EKS clusters (untested). I wrote an issue here that condenses the problems I encountered on non-EKS cluster with aws-load-balancer-controller and this is one of them.

[radupopa369]

This is frustrating to learn that such an important feature is not automated today, almost 5 years later