-
Notifications
You must be signed in to change notification settings - Fork 9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104
Comments
+1 I went through the example here: In which
I really wish AWS offered an easier way to get the thumbprint of the certificate. |
After some searching it looks like this could solve our issue if implemented: |
In the mean time here is a quick hack to get around it, not ideal but tested working echo | openssl s_client -connect oidc.eks.us-west-2.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' locals {
eks-oidc-thumbprint = "$OUTPUT_FROM_ABOVE"
}
resource "aws_iam_openid_connect_provider" "eks-oidc" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [local.eks-oidc-thumbprint]
url = "${aws_eks_cluster.cluster.identity.0.oidc.0.issuer}"
} |
There is an automated way that I received from kubernetes slack which is using external source:
|
I personally think this thumbprint thing should live in terraform-provider-aws repo.
But aws_iam_openid_connect_provider needs it to be like (yes lowercase)
I don't know if this specific transform is general enough to live in Terraform TLS provider. |
For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script. thumbprint.sh #!/bin/bash
THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON terraform.tf data "external" "thumbprint" {
program = ["thumbprint.sh", data.aws_region.current.name]
}
resource "aws_iam_openid_connect_provider" "this" {
client_id_list = ["sts.amazonaws.com"]
thumbprint_list = [data.external.thumbprint.result.thumbprint]
url = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
} |
@dogzzdogzz thanks for your solution! Unfortunately i tried it without any luck. It is giving me an incorrect thumbprint. From my understanding of the aws tutorial we need to extract only the last certificate between Then we can give it to |
Yep, that produces a non-working thumbprint. Here's my horrible oneliner to produce a working thumbprint: Edit: this is for OSX, for GNU/Linux replace |
Thanks! I combined the two solutions to get a working terraform resource |
So the solutions above don't work for our setup, we use an https proxy and thus the CA chain won't be correct if I got through that. I can tunnel through a bastion, but am I missing something here, that CA root will be mostly static for each AWS region, no? Wouldn't it be viable to just keep the fingerprints for each region in a TF map until a better solution presents itself? |
@chiefy you don't even need a map - it looks like the root CA is the same for the two regions I tried, and I imagine that's the case for all the regions. Setting thumbprints to |
@willthames thanks for confirming my sanity. |
For anybody who is running into issues with differing or inconsistent thumbprints, this might help ...
☝️ This is the thumbprint you want. But, if you are performing this same operation from outside the cluster, you will likely receive something like:
HTH |
@dayglojesus yes, it helps, but still there is a problem with automating this. If we need to run command from inside Pod to get thumbprint required to automate IAM role policies for ServiceAccounts makes it impossible to work :-/ |
@willthames well, this will result in constant drift because the thumbprint needs to be lowercase (at least that's my experience). I merged my previous ugly one-liner with the suggestions here and made this (for tf v.0.11.x):
It seems that this works perfectly fine for the |
@rastakajakwanna yes, you're quite right, I think I wrote my comment after realising that it worked, but before realising about the drift you mention. But I'm literally just hardcoding the (lowercase) value into a variable. I don't see the benefit of having terraform run a script to generate a value that so far is consistent across all regions, and given it's a root CA, likely will remain so for a decade. |
This works with terraform cloud
|
Should this issue be marked as a bug? The terraform provider documentation is misleading, as an empty |
@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect. |
Fair enough 😄 , I've raised it with AWS, but it feels it's a sane behavior not to expose the thumbprint through the API response on their side, which is probably why the AWS Console auto populate it for convenience. Let's follow #10217 |
If anyone needs to run @mzupan example on macOS/BSD/others, please consider that
Thanks @mzupan ❤️ |
@dayglojesus @Grejeru I think I found the root issue for different certificates from different clients: It depends on the implementation/version of the Other implementations like LibreSSL (on Mac) or CodeBuild/AmazonLinux 2 (which use The following command should work independently from OpenSSL versions/implemenentations:
[1] https://www.openssl.org/docs/man1.1.1/man1/openssl-s_client.html |
I have an updated version that tests for a env var used by tf cloud
|
If your openssl's x509 fingerprint behaviour defaults to something other than sha1 you may get invalid fingerprints:
Being explicit fixes this:
|
The thumbprint even seems to be the same
|
This adds a list of server certificate thumbprints as a computed attribute to both the aws_eks_cluster resource and aws_eks_cluster data source. It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html in go. The happy path acceptance tests have been updated. Fixes hashicorp#10104
@mzupan I had to modify this answer to work for me, as I am using it in a module: |
You can use |
You can generate fingerprint and set in your thumbprint_list follow how to this link: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html Those are all CA's fingerprint by region:
You can define your variable like this: thumbprint_list = ["3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8","59ADE3A3A6039BBA092E920FEE413466493F409C","3FB881BCACBD420928168C739B07EEF47555946B","7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B","CAB073498D7558FEC3B2C414C006ACBA30805431","2EAFC197C15CDEE5426BFD4D27D3321A685F3B78","B715DC079832DA5FC1D4706515BE48BE79A1C871","CB454452665937052981CA118417B7A162A25F54","1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1","F719C49FEA86549E159818880E392C1570C953B6","0148872FA02F3A7D6B38AA88FA5397228B28E08B","9884072430220E6253011B88F940E4F20F53D0CC","598ADECB9A3E6CC70AA53D64BD5EE4704300382A","750B948515281953BC6F3D717A1E1654ECBFA852","89BABC6D46502653516CC0BA38B14A2B7864D161","63966130761608209718C5045CFFB4856FB53976"] command for get fingerprint : echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'
|
@romaryoricardo I haven't test this yet, but I recommend @mzupan (or @cippaciong for macos) workaround over yours. I am not sure why this happens, but the fingerprints generated using your command are not the same as the ones generated by @mzupan commands (I am going to throw the guess that multiple certificates are being spit out for each command, but only 1 is processed, in @mzupan case is the most recent one, and in your case is the oldest one, although i can be completely wrong). It doesn't mean your solutions shouldn't work, but if we inspect the certificate of the fingerprint that your command spits, we can see that this certificate is about to expire (check the validity, it expires september 2020):
whereas in the other, it expires on 2034. |
For those who are not happy with dirty tricks in shell script:
For alpine docker images it enough to install:
For python:3 docker image:
|
#10104 (comment) used to be show longer certificate, which expire 2034.
{"thumbprint": "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"} However I've ran same script and it shows shorter certificate, it expires on Sep 2020. (And is match to certificate show on browser.)
{"thumbprint": "b715dc079832da5fc1d4706515be48be79a1c871"} Certificate Chain
|
@guitarrapc this is a correct value for your region. I feel like this is an issue that can now be closed. There are numerous solutions suggested that will solve problems with OIDC provider. |
Oof, I just created a new cluster today and it has a new cert for
|
@nickatsegment I'm thinking this is as you've made the same mistake I made and are using the "leaf" server cert which gets updated owing to it's short expiry. Instead use the root Certificate Authority cert which expires in 2034 or similar. Switch to use the commands above which have |
Taken from a number of sources but with explicit help from this article https://medium.com/@michael.kandelaars/did-your-eks-iam-service-account-roles-break-today-2ea50c869aee, this thread & of course Stackoverflow for bash-fu, I've got this script which grabs the root ca thumbprint. Not the prettiest, but function over form right? #!/usr/bin/env bash
echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | awk '/-----BEGIN/{f="cert."(n++)} f{print>f} /-----END/{f=""}'
certificates=()
for c in cert.*; do
certificates+=($(openssl x509 <$c -noout -fingerprint))
done
rm cert.*
thumbprint=$(echo ${certificates[${#certificates[@]}-1]} | sed 's/://g' | awk -F= '{print tolower($2)}')
thumbprint_json="{\"thumbprint\": \"${thumbprint}\"}"
echo $thumbprint_json
|
@tyrken ah yep, you're right. Thanks for the pointer. Here's hoping that TLS provider PR lands sometime in the next century so we can all dispense with the janky awk scripts |
My external for this external/thumbprint
Use it like this:
For eu-central-1 region it returns fingerprint for this cert:
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
Description
New or Affected Resource(s)
Currently I can specify the following:
There is no way to retrieve thumbprint for that OIDC provider using terraform.
Note that if you create the same OIDC provider in the console, it will automatically populate the thumbprint which is required for EKS service accounts to assume correct IAM Role.
References
Current way of getting thumbprint is documented here -> https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2
The text was updated successfully, but these errors were encountered: