-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DNS01 Challenge via AWS Route53 on AWS EKS 1.15 not working #3079
Comments
Seems this part is missing: https://cert-manager.io/docs/configuration/acme/dns01/route53/#iam-role-trust-policy |
Hello, |
Hi, I am seeing something similar. Error when describing challenge resources
ClusterIssuer
Trust Relationship
Service AccountI've annotated the cert-manager service account as described here.
Security ContextI also tried updating the security context as described here.
Environment
|
We are also having the same issue as described above by @JanisOrlovs and @akoudal. Wanted to add the context that my setup is
Install method:
values.yaml:
i tried removing the condition on the trust relationship too but that didn't help get the role assumed. i too am getting the error described above
|
I got it working. I've described my installation procedure below. Hope can help solve your issues too @johndietz and @JanisOrlovs . CertManagerThis section describes how to install and setup cert-manager on an AWS EKS cluster. It will integrate with Route53 DNS for issuing DNS-based certificate challenges. IAM Open ID Connect provider for the clusterThis is required for associating k8s Service Accounts with AWS IAM roles and policies. This is considered more secure than allowing the worker nodes to access AWS services (e.g. S3 Buckets, Route53 DNS, etc.). eksctl utils associate-iam-oidc-provider --cluster name-of-the-cluster More info about the command above can be found here. AWS IAM PolicyCreate an AWS IAM policy named AllowExternalDNSUpdates and attach the following permissions to it: {
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "route53:ChangeResourceRecordSets",
"Resource": "arn:aws:route53:::hostedzone/*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"route53:GetChange",
"route53:ListHostedZones",
"route53:ListResourceRecordSets",
"route53:ListHostedZonesByName"
],
"Resource": "*"
}
]
} Less can probably do but the above works well if you want to use the same policy with e.g. external-dns. Check the cert-manager docs if you wish to limit the policy to specific Route53 resources. Create the AWS IAM RoleCreate an IAM Service Account thats allowed to perform DNS updates (Route53). This is required in order to use DNS-based certificate challenges. aws iam create-role --role-name cert-manager Attach the AllowExternalDNSUpdates policy to the roleThis assumes that a policy named AllowExternalDNSUpdates already exists at arn:aws:iam::XXXXXXXXXXXX:policy/AllowExternalDNSUpdates (created in an earlier step) aws iam attach-role-policy --policy-arn arn:aws:iam::XXXXXXXXXXXX:policy/AllowExternalDNSUpdates --role-name cert-manager Trust relationshipAssign a trust relationship to the newly created IAM role in AWS web console, allowing you to map kubernetes service accounts to AWS IAM roles. It should look something like this, with the values in <value> replaced accordingly: {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/oidc.eks.eu-north-1.amazonaws.com/id/<eks-hash>"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-north-1.amazonaws.com/id/<eks-hash>:sub": "system:serviceaccount:cert-manager:cert-manager"
}
}
}
]
} You can find the value at Principal.Federated at https://console.aws.amazon.com/iam/home?region=eu-north-1#/providers (replace eu-north-1 with your region). You can extract the <aws-account-id> and <eks-hash> from that string too. Cert manager custom valuesCreate a file called cert-manager.values.yaml securityContext:
enabled: "true"
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::<aws-account-id>:role/cert-manager
Note the securityContext part which is a fix if you experience the issue described in issue #2147. Install CertManager from Helm templateStart by checking that you have the correct <aws-account-id> in cert-manager.values.yaml, then run the command below: helm template cert-manager jetstack/cert-manager \
--namespace cert-manager \
--version v0.15.2 \
--set installCRDs=true \
-f cert-manager.values.yaml | kubectl apply -f - ClusterIssuerCreate a ClusterIssuer which can be used to issue Lets Encrypt certificates in all namespaces. Save the following yaml into a file called letsencrypt.clusterissuer.yaml apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: your@email.com
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: letsencrypt-pk
# Use DNS as challenge solver
solvers:
- dns01:
route53:
region: eu-north-1 Apply the manifest to your cluster: kubectl apply -f letsencrypt.clusterissuer.yaml |
@akoudal i wanted to thank you for the time you took to document your walkthough of these items and that you got past the error. i regret to say that i've followed these steps pretty much to the letter, and i continue to get the same issue. differences between our setups are that i'm still using the staging server for acme, though i trust that's not in play. i'm also in us-west-2 and i'm not installing crd with the helm chart, but rather deploying those separately. i should also call out that being on eks 1.14, i'm stuck on the legacy CRDs, perhaps something might be in play there. |
What does your ClusterIssuer look like @johndietz? I was seeing some AssumeRole errors until I removed the "role" entry from my dns01 solver. |
@akoudal the script i'd hacked together to iterate apparently missed on the directory that houses my clusterissuers. once i applied them with your changes the sts issue went away. i really can't thank you enough man. i'll open a PR against their route53 docs to pay your assistance forward. |
Awesome. Glad you got it working @johndietz. |
the only route53 dn01 clusterissuer example prior to this change specifies a role in each solver example, however in a single-account eks setup where the role is instead specified on the serviceaccount, including the same role on the clusterissuer causes sts errors and prevents the dns management. i welcome any feedback on the comments i've provided in this eks clusterissuer example. see discussion in cert-manager/cert-manager#3079 for more context on this issue.
the only route53 dn01 clusterissuer example prior to this change specifies a role in each solver example, however in a single-account eks setup where the role is instead specified on the serviceaccount, including the same role on the clusterissuer causes sts errors and prevents the dns management. i welcome any feedback on the comments i've provided in this eks clusterissuer example. see discussion in cert-manager/cert-manager#3079 for more context on this issue. Signed-off-by: John Dietz <jdietz@jdietz.fios-router.home>
Essentially, When the |
Thanks for clarifying, @anarsen. Took a while to figure out how to force it to use the eks.amazonaws.com/role-arn: arn:aws:iam:::role/cert-manager from the annotation in stead of an assumed role. |
It looks like this issue has been resolved. I'll close it for now, if anybody feels otherwise, feel free to reopen it. /close |
@hzhou97: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Hi folks, On AWS EKS: v1.17.9 I'm hitting the error: The IAM role is not being accessed/used in AWS, and the IRSA token is mounted on the pods along with the I've tried Any input would be greatly appreciated! |
Update: tried v0.15.2 with a It seems that something may have broken in between releases or something w.r.t IRSA was not accurately captured in the docs. Follow up: v1.0.3 of the helm chart is working with a |
I've got the same issue as @metral with |
I was running into the same issue, I added this to my values.yaml extraArgs:
- --issuer-ambient-credentials It appears that by default the namespace Issuers do not allow using "ambient" credentials, this let's them. |
This is lowkey messy - devs update docs pls |
This should help reduce the amount of time people might waste trying to figure out how to resolve the following error: ``` error instantiating route53 challenge solver: unable to construct route53 provider: empty credentials; perhaps you meant to enable ambient credentials? ``` A couple of related bug reports: * cert-manager/cert-manager#3009 * cert-manager/cert-manager#3079
@ZeChArtiahSaher and others: If you have time, please review and comment on: |
This should help reduce the amount of time people might waste trying to figure out how to resolve the following error: ``` error instantiating route53 challenge solver: unable to construct route53 provider: empty credentials; perhaps you meant to enable ambient credentials? ``` A couple of related bug reports: * cert-manager/cert-manager#3009 * cert-manager/cert-manager#3079
The documentation update still doesn't reflect the update. Can someone please check and do the needful ? Thank you :) |
How did you resolve the issue with that you received your error from the start? // Tobias |
These lines of code + this issue made me realise I need the Thanks @heschlie |
Describe the bug:
When doing Let's Encrypt validation via Route53 running on EKS 1.15 with IRSA getting AccessDenied
Expected behaviour:
Issued LE cert via DNS01 validation
Steps to reproduce the bug:
Create following cluster-issuer and ingress
ClusterIssuer object:
Ingress object:
Anything else we need to know?:
IRSA working properly, tested. Same role for R53 working good from CLI
IAM policy(veeery broad):
EDIT
Thrust relationship:
Plus trust relationships working as needed (tested with pod)
SA object:
Environment details::
/kind bug
The text was updated successfully, but these errors were encountered: