Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104

Open
marcincuber opened this issue Sep 13, 2019 · 17 comments · May be fixed by #10217

Comments

@marcincuber
Copy link

@marcincuber marcincuber commented Sep 13, 2019

Description

New or Affected Resource(s)

Currently I can specify the following:

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

There is no way to retrieve thumbprint for that OIDC provider using terraform.

Note that if you create the same OIDC provider in the console, it will automatically populate the thumbprint which is required for EKS service accounts to assume correct IAM Role.

References

Current way of getting thumbprint is documented here -> https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2

@gkrizek

This comment has been minimized.

Copy link
Contributor

@gkrizek gkrizek commented Sep 18, 2019

+1

I went through the example here:
https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#enabling-iam-roles-for-service-accounts

In which thumbprint_list is an empty list. Everything worked until I tried to actually do an AWS operation:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I really wish AWS offered an easier way to get the thumbprint of the certificate.

@gkrizek

This comment has been minimized.

Copy link
Contributor

@gkrizek gkrizek commented Sep 19, 2019

After some searching it looks like this could solve our issue if implemented:
terraform-providers/terraform-provider-tls#52

wdullaer added a commit to wdullaer/terraform-provider-aws that referenced this issue Sep 24, 2019
This adds a list of server certificate thumbprints as a computed attribute
to both the aws_eks_cluster resource and aws_eks_cluster data source.
It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
in go.
The happy path acceptance tests have been updated.

Fixes terraform-providers#10104
@zzh8829

This comment has been minimized.

Copy link

@zzh8829 zzh8829 commented Sep 24, 2019

In the mean time here is a quick hack to get around it, not ideal but tested working

echo | openssl s_client -connect oidc.eks.us-west-2.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'
locals {
  eks-oidc-thumbprint = "$OUTPUT_FROM_ABOVE"
}

resource "aws_iam_openid_connect_provider" "eks-oidc" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [local.eks-oidc-thumbprint]
  url             = "${aws_eks_cluster.cluster.identity.0.oidc.0.issuer}"
}
wdullaer added a commit to wdullaer/terraform-provider-aws that referenced this issue Sep 24, 2019
This adds a list of server certificate thumbprints as a computed attribute
to both the aws_eks_cluster resource and aws_eks_cluster data source.
It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
in go.
The happy path acceptance tests have been updated.

Fixes terraform-providers#10104
wdullaer added a commit to wdullaer/terraform-provider-aws that referenced this issue Sep 24, 2019
This adds a list of server certificate thumbprints as a computed attribute
to both the aws_eks_cluster resource and aws_eks_cluster data source.
It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
in go.
The happy path acceptance tests have been updated.

Fixes terraform-providers#10104
@bflad bflad removed the needs-triage label Sep 24, 2019
@marcincuber

This comment has been minimized.

Copy link
Author

@marcincuber marcincuber commented Sep 24, 2019

There is an automated way that I received from kubernetes slack which is using external source:

### External cli kubergrunt
data "external" "thumb" {
  program = ["kubergrunt", "eks", "oidc-thumbprint", "--issuer-url", aws_eks_cluster.cluster.identity.0.oidc.0.issuer]
}
### OIDC config
resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumb.result.thumbprint]
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}
@zzh8829

This comment has been minimized.

Copy link

@zzh8829 zzh8829 commented Sep 24, 2019

I personally think this thumbprint thing should live in terraform-provider-aws repo.
SHA1 finger print are usually in the form of

SHA1 Fingerprint=63:96:61:30:76:16:08:20:97:18:C5:04:5C:FF:B4:85:6F:B5:39:76

But aws_iam_openid_connect_provider needs it to be like (yes lowercase)

63966130761608209718c5045cffb4856fb53976

I don't know if this specific transform is general enough to live in Terraform TLS provider.

@dogzzdogzz

This comment has been minimized.

Copy link

@dogzzdogzz dogzzdogzz commented Sep 25, 2019

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}
@matteosilv

This comment has been minimized.

Copy link

@matteosilv matteosilv commented Sep 30, 2019

@dogzzdogzz thanks for your solution! Unfortunately i tried it without any luck. It is giving me an incorrect thumbprint.

From my understanding of the aws tutorial we need to extract only the last certificate between BEGIN CERTIFICATE and END CERTIFICATE, that one is the certificate of the root CA.

Then we can give it to openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' to extract the thumbprint

@JamiKarvanen

This comment has been minimized.

Copy link

@JamiKarvanen JamiKarvanen commented Sep 30, 2019

Yep, that produces a non-working thumbprint. Here's my horrible oneliner to produce a working thumbprint:
echo | openssl s_client -servername oidc.eks.${REGION}.amazonaws.com -showcerts -connect oidc.eks.${REGION}.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

@matteosilv

This comment has been minimized.

Copy link

@matteosilv matteosilv commented Oct 1, 2019

Thanks! I combined the two solutions to get a working terraform resource

@chiefy

This comment has been minimized.

Copy link

@chiefy chiefy commented Oct 11, 2019

So the solutions above don't work for our setup, we use an https proxy and thus the CA chain won't be correct if I got through that. I can tunnel through a bastion, but am I missing something here, that CA root will be mostly static for each AWS region, no? Wouldn't it be viable to just keep the fingerprints for each region in a TF map until a better solution presents itself?

@willthames

This comment has been minimized.

Copy link

@willthames willthames commented Oct 23, 2019

@chiefy you don't even need a map - it looks like the root CA is the same for the two regions I tried, and I imagine that's the case for all the regions.

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@chiefy

This comment has been minimized.

Copy link

@chiefy chiefy commented Oct 23, 2019

@willthames thanks for confirming my sanity.

@dayglojesus

This comment has been minimized.

Copy link

@dayglojesus dayglojesus commented Oct 30, 2019

For anybody who is running into issues with differing or inconsistent thumbprints, this might help ...

When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

subject=CN = oidc.eks.us-west-2.amazonaws.com
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

☝️ This is the thumbprint you want.

But, if you are performing this same operation from outside the cluster, you will likely receive something like:

subject=/CN=*.execute-api.us-west-2.amazonaws.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon

HTH

@Grejeru

This comment has been minimized.

Copy link

@Grejeru Grejeru commented Oct 31, 2019

@dayglojesus yes, it helps, but still there is a problem with automating this. If we need to run command from inside Pod to get thumbprint required to automate IAM role policies for ServiceAccounts makes it impossible to work :-/
Nevertheless thx, as it solve my issue why script generated thumbprint didn't worked and had to find it other way. Which is even funnier, there are more then 1 cert, to be exact 4 of them and with this list I got it working.

@dayglojesus

This comment has been minimized.

Copy link

@dayglojesus dayglojesus commented Oct 31, 2019

@Grejeru I agree, this shouldn't be a hard requirement, but per @chiefy's comments ... You should be able to construct a map of the correct thumbs, one for each required region.

@rastakajakwanna

This comment has been minimized.

Copy link

@rastakajakwanna rastakajakwanna commented Nov 7, 2019

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@willthames well, this will result in constant drift because the thumbprint needs to be lowercase (at least that's my experience).
However, all our eu regions have the same thumbprint. I tried the script from outside of VPC as well as from Gitlab runner in EKS and I've got the same results/thumbprint.

I merged my previous ugly one-liner with the suggestions here and made this (for tf v.0.11.x):

<module_name>/bin/get_thumbprint.sh

#!/bin/bash
set -e 

XR="${1:-eu-west-1}"
XT=`mktemp`
XXT=`mktemp`

function cleanup {
  rm -f ${XT} ${XXT}
}

trap cleanup SIGHUP SIGINT SIGTERM EXIT

THUMBPRINT=$(echo QUIT | openssl s_client -showcerts -connect oidc.eks.${XR}.amazonaws.com:443 2>/dev/null > ${XT}; cat ${XT} | sed -n '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | tac | awk '/-----BEGIN CERTIFICATE-----/ {exit} 1' > ${XXT} && echo '-----BEGIN CERTIFICATE-----' >> ${XXT} && tac ${XXT} > ${XT}; openssl x509 -in ${XT} -fingerprint -noout | sed -r 's|.*+?=(.*)|\1|g' | sed 's|:||g' | awk '{print tolower($1)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

<module_name>/data.tf

data "aws_region" "current" {}
data "external" "thumbprint" {
  program = ["${path.module}/bin/get_thumbprint.sh", "${data.aws_region.current.name}"]
}

<module_name>/main.tf

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["${data.external.thumbprint.result.thumbprint}"]
  url             = "${data.aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

It seems that this works perfectly fine for the kube-system aws-node service account. At least as much as I can say after two days....

@willthames

This comment has been minimized.

Copy link

@willthames willthames commented Nov 7, 2019

@rastakajakwanna yes, you're quite right, I think I wrote my comment after realising that it worked, but before realising about the drift you mention.

But I'm literally just hardcoding the (lowercase) value into a variable. I don't see the benefit of having terraform run a script to generate a value that so far is consistent across all regions, and given it's a root CA, likely will remain so for a decade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.