Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retrieve OIDC provider thumbprint not populating which is required for EKS service accounts #10104

Closed
marcincuber opened this issue Sep 13, 2019 · 39 comments
Labels
enhancement service/iam

Comments

@marcincuber
Copy link

@marcincuber marcincuber commented Sep 13, 2019

Description

New or Affected Resource(s)

Currently I can specify the following:

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = []
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

There is no way to retrieve thumbprint for that OIDC provider using terraform.

Note that if you create the same OIDC provider in the console, it will automatically populate the thumbprint which is required for EKS service accounts to assume correct IAM Role.

References

Current way of getting thumbprint is documented here -> https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html#thumbstep2

@marcincuber marcincuber added the enhancement label Sep 13, 2019
@ghost ghost added the service/iam label Sep 13, 2019
@github-actions github-actions bot added the needs-triage label Sep 13, 2019
@gkrizek
Copy link
Contributor

@gkrizek gkrizek commented Sep 18, 2019

+1

I went through the example here:
https://www.terraform.io/docs/providers/aws/r/eks_cluster.html#enabling-iam-roles-for-service-accounts

In which thumbprint_list is an empty list. Everything worked until I tried to actually do an AWS operation:

An error occurred (InvalidIdentityToken) when calling the AssumeRoleWithWebIdentity operation: OpenIDConnect provider's HTTPS certificate doesn't match configured thumbprint

I really wish AWS offered an easier way to get the thumbprint of the certificate.

@gkrizek
Copy link
Contributor

@gkrizek gkrizek commented Sep 19, 2019

After some searching it looks like this could solve our issue if implemented:
hashicorp/terraform-provider-tls#52

@zzh8829
Copy link

@zzh8829 zzh8829 commented Sep 24, 2019

In the mean time here is a quick hack to get around it, not ideal but tested working

echo | openssl s_client -connect oidc.eks.us-west-2.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'
locals {
  eks-oidc-thumbprint = "$OUTPUT_FROM_ABOVE"
}

resource "aws_iam_openid_connect_provider" "eks-oidc" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [local.eks-oidc-thumbprint]
  url             = "${aws_eks_cluster.cluster.identity.0.oidc.0.issuer}"
}

@bflad bflad removed the needs-triage label Sep 24, 2019
@marcincuber
Copy link
Author

@marcincuber marcincuber commented Sep 24, 2019

There is an automated way that I received from kubernetes slack which is using external source:

### External cli kubergrunt
data "external" "thumb" {
  program = ["kubergrunt", "eks", "oidc-thumbprint", "--issuer-url", aws_eks_cluster.cluster.identity.0.oidc.0.issuer]
}
### OIDC config
resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumb.result.thumbprint]
  url             = aws_eks_cluster.cluster.identity.0.oidc.0.issuer
}

@zzh8829
Copy link

@zzh8829 zzh8829 commented Sep 24, 2019

I personally think this thumbprint thing should live in terraform-provider-aws repo.
SHA1 finger print are usually in the form of

SHA1 Fingerprint=63:96:61:30:76:16:08:20:97:18:C5:04:5C:FF:B4:85:6F:B5:39:76

But aws_iam_openid_connect_provider needs it to be like (yes lowercase)

63966130761608209718c5045cffb4856fb53976

I don't know if this specific transform is general enough to live in Terraform TLS provider.

@dogzzdogzz
Copy link

@dogzzdogzz dogzzdogzz commented Sep 25, 2019

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

@matteosilv
Copy link

@matteosilv matteosilv commented Sep 30, 2019

@dogzzdogzz thanks for your solution! Unfortunately i tried it without any luck. It is giving me an incorrect thumbprint.

From my understanding of the aws tutorial we need to extract only the last certificate between BEGIN CERTIFICATE and END CERTIFICATE, that one is the certificate of the root CA.

Then we can give it to openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}' to extract the thumbprint

@JamiKarvanen
Copy link

@JamiKarvanen JamiKarvanen commented Sep 30, 2019

Yep, that produces a non-working thumbprint. Here's my horrible oneliner to produce a working thumbprint:
echo | openssl s_client -servername oidc.eks.${REGION}.amazonaws.com -showcerts -connect oidc.eks.${REGION}.amazonaws.com:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

Edit: this is for OSX, for GNU/Linux replace tail -r with tac

@matteosilv
Copy link

@matteosilv matteosilv commented Oct 1, 2019

Thanks! I combined the two solutions to get a working terraform resource

@chiefy
Copy link

@chiefy chiefy commented Oct 11, 2019

So the solutions above don't work for our setup, we use an https proxy and thus the CA chain won't be correct if I got through that. I can tunnel through a bastion, but am I missing something here, that CA root will be mostly static for each AWS region, no? Wouldn't it be viable to just keep the fingerprints for each region in a TF map until a better solution presents itself?

@willthames
Copy link

@willthames willthames commented Oct 23, 2019

@chiefy you don't even need a map - it looks like the root CA is the same for the two regions I tried, and I imagine that's the case for all the regions.

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@chiefy
Copy link

@chiefy chiefy commented Oct 23, 2019

@willthames thanks for confirming my sanity.

@dayglojesus
Copy link

@dayglojesus dayglojesus commented Oct 30, 2019

For anybody who is running into issues with differing or inconsistent thumbprints, this might help ...

When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

subject=CN = oidc.eks.us-west-2.amazonaws.com
issuer=C = US, O = Amazon, OU = Server CA 1B, CN = Amazon

☝️ This is the thumbprint you want.

But, if you are performing this same operation from outside the cluster, you will likely receive something like:

subject=/CN=*.execute-api.us-west-2.amazonaws.com
issuer=/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon

HTH

@Grejeru
Copy link

@Grejeru Grejeru commented Oct 31, 2019

@dayglojesus yes, it helps, but still there is a problem with automating this. If we need to run command from inside Pod to get thumbprint required to automate IAM role policies for ServiceAccounts makes it impossible to work :-/
Nevertheless thx, as it solve my issue why script generated thumbprint didn't worked and had to find it other way. Which is even funnier, there are more then 1 cert, to be exact 4 of them and with this list I got it working.

@dayglojesus
Copy link

@dayglojesus dayglojesus commented Oct 31, 2019

@Grejeru I agree, this shouldn't be a hard requirement, but per @chiefy's comments ... You should be able to construct a map of the correct thumbs, one for each required region.

@rastakajakwanna
Copy link

@rastakajakwanna rastakajakwanna commented Nov 7, 2019

Setting thumbprints to ["9E99A48A9960B14926BB7F3B02E22DA2B0AB7280"] should suffice :)

@willthames well, this will result in constant drift because the thumbprint needs to be lowercase (at least that's my experience).
However, all our eu regions have the same thumbprint. I tried the script from outside of VPC as well as from Gitlab runner in EKS and I've got the same results/thumbprint.

I merged my previous ugly one-liner with the suggestions here and made this (for tf v.0.11.x):

<module_name>/bin/get_thumbprint.sh

#!/bin/bash
set -e 

XR="${1:-eu-west-1}"
XT=`mktemp`
XXT=`mktemp`

function cleanup {
  rm -f ${XT} ${XXT}
}

trap cleanup SIGHUP SIGINT SIGTERM EXIT

THUMBPRINT=$(echo QUIT | openssl s_client -showcerts -connect oidc.eks.${XR}.amazonaws.com:443 2>/dev/null > ${XT}; cat ${XT} | sed -n '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p' | tac | awk '/-----BEGIN CERTIFICATE-----/ {exit} 1' > ${XXT} && echo '-----BEGIN CERTIFICATE-----' >> ${XXT} && tac ${XXT} > ${XT}; openssl x509 -in ${XT} -fingerprint -noout | sed -r 's|.*+?=(.*)|\1|g' | sed 's|:||g' | awk '{print tolower($1)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

<module_name>/data.tf

data "aws_region" "current" {}
data "external" "thumbprint" {
  program = ["${path.module}/bin/get_thumbprint.sh", "${data.aws_region.current.name}"]
}

<module_name>/main.tf

resource "aws_iam_openid_connect_provider" "cluster" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = ["${data.external.thumbprint.result.thumbprint}"]
  url             = "${data.aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

It seems that this works perfectly fine for the kube-system aws-node service account. At least as much as I can say after two days....

@willthames
Copy link

@willthames willthames commented Nov 7, 2019

@rastakajakwanna yes, you're quite right, I think I wrote my comment after realising that it worked, but before realising about the drift you mention.

But I'm literally just hardcoding the (lowercase) value into a variable. I don't see the benefit of having terraform run a script to generate a value that so far is consistent across all regions, and given it's a root CA, likely will remain so for a decade.

@mzupan
Copy link
Contributor

@mzupan mzupan commented Dec 13, 2019

This works with terraform cloud

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON


data "external" "thumbprint" {
  program = [format("%s/bin/get_thumbprint.sh", path.module), data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = module.eks.cluster_oidc_issuer_url
}


@jujugrrr
Copy link

@jujugrrr jujugrrr commented Jan 3, 2020

Should this issue be marked as a bug? The terraform provider documentation is misleading, as an empty thumbprint doesn't allow to use EKS IAM with service account.

@marcincuber
Copy link
Author

@marcincuber marcincuber commented Jan 3, 2020

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

@jujugrrr
Copy link

@jujugrrr jujugrrr commented Jan 3, 2020

@jujugrrr I don't think this is a bug. Terraform is simply missing a way to retrieve the thumbprint. I do agree that documentation is not perfect.

Fair enough 😄 , I've raised it with AWS, but it feels it's a sane behavior not to expose the thumbprint through the API response on their side, which is probably why the AWS Console auto populate it for convenience.

Let's follow #10217

@cippaciong
Copy link

@cippaciong cippaciong commented Jan 16, 2020

If anyone needs to run @mzupan example on macOS/BSD/others, please consider that tac is not available there by default and you need to either install coreutils or replace it with tail -r, which is what I did:

#/bin/bash

THUMBPRINT=$(echo | \
    openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | \
    tail -r | \
    sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | \
    tail -r | \
    openssl x509 -fingerprint -noout | \
    sed 's/://g' | awk -F= '{print tolower($2)}')

THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"

echo $THUMBPRINT_JSON

Thanks @mzupan ❤️

@s0enke
Copy link

@s0enke s0enke commented Jan 23, 2020

When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the cluster (from one of your EKS workers), you get a cert like:
When running openssl s_client -servername oidc.eks.${REGION}.amazonaws.com etc. from "inside" the pod, you get a cert like:

@dayglojesus @Grejeru I think I found the root issue for different certificates from different clients:

It depends on the implementation/version of the openssl command. In OpenSSL (since 1.1?) the connect will automatically send an servername with the connection (Server Name Indication). See [1]

Other implementations like LibreSSL (on Mac) or CodeBuild/AmazonLinux 2 (which use OpenSSL 1.0.2) don't automatically set the servername so we have to provide it.

The following command should work independently from OpenSSL versions/implemenentations:

openssl s_client -connect oidc.eks.eu-central-1.amazonaws.com:443 -servername oidc.eks.eu-central-1.amazonaws.com

[1] https://www.openssl.org/docs/man1.1.1/man1/openssl-s_client.html

@mzupan
Copy link
Contributor

@mzupan mzupan commented Jan 23, 2020

@cippaciong

I have an updated version that tests for a env var used by tf cloud

#!/bin/bash



if [[ -z "${ATLAS_WORKSPACE_SLUG}" ]]; then
  APP="tail -r"
else
  APP="tac"
fi

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | ${APP} | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | ${APP} | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

@christian-marie
Copy link

@christian-marie christian-marie commented Mar 3, 2020

If your openssl's x509 fingerprint behaviour defaults to something other than sha1 you may get invalid fingerprints:

Member must satisfy constraint: [Member must have length less than or equal to 40, Member must have length greater than or equal to 40]

Being explicit fixes this:

#!/bin/sh
# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

THUMBPRINT=$(echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | tac | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tac | openssl x509 -fingerprint -sha1 -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

@groodt
Copy link

@groodt groodt commented Mar 5, 2020

The thumbprint even seems to be the same cn-northwest-1

echo | openssl s_client -servername oidc.eks.cn-northwest-1.amazonaws.com.cn -showcerts -connect oidc.eks.cn-northwest-1.amazonaws.com.cn:443 2>&- | tail -r | sed -n '/-----END CERTIFICATE-----/,/-----BEGIN CERTIFICATE-----/p; /-----BEGIN CERTIFICATE-----/q' | tail -r | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}'

wdullaer added a commit to wdullaer/terraform-provider-aws that referenced this issue Mar 25, 2020
This adds a list of server certificate thumbprints as a computed attribute
to both the aws_eks_cluster resource and aws_eks_cluster data source.
It implements the instructions from https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
in go.
The happy path acceptance tests have been updated.

Fixes hashicorp#10104
@scardena
Copy link

@scardena scardena commented May 21, 2020

For anyone prefer not to install kubergrunt, I merge both @zzh8829 and @marcincuber workaround to make terraform automatically retrieve thumbprint from external script.

thumbprint.sh

#!/bin/bash

THUMBPRINT=$(echo | openssl s_client -connect oidc.eks.$1.amazonaws.com:443 2>&- | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print tolower($2)}')
THUMBPRINT_JSON="{\"thumbprint\": \"${THUMBPRINT}\"}"
echo $THUMBPRINT_JSON

terraform.tf

data "external" "thumbprint" {
  program = ["thumbprint.sh", data.aws_region.current.name]
}

resource "aws_iam_openid_connect_provider" "this" {
  client_id_list  = ["sts.amazonaws.com"]
  thumbprint_list = [data.external.thumbprint.result.thumbprint]
  url             = "${aws_eks_cluster.this.identity.0.oidc.0.issuer}"
}

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

@Grejeru
Copy link

@Grejeru Grejeru commented May 22, 2020

@mzupan I had to modify this answer to work for me, as I am using it in a module:
program = ["${path.module}/thumbprint.sh", var.aws_region]

You can use data "aws_region" "current" {} in the module too, it assumes the region in which Your provider works, don't need to pass aws_region into module then.

@romaryoricardo
Copy link

@romaryoricardo romaryoricardo commented May 23, 2020

You can generate fingerprint and set in your thumbprint_list follow how to this link: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html

Those are all CA's fingerprint by region:

eu-north-1
3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8
ap-south-1
59ADE3A3A6039BBA092E920FEE413466493F409C
eu-west-3
3FB881BCACBD420928168C739B07EEF47555946B
eu-west-2
7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B
eu-west-1
CAB073498D7558FEC3B2C414C006ACBA30805431
ap-northeast-2
2EAFC197C15CDEE5426BFD4D27D3321A685F3B78
ap-northeast-1
B715DC079832DA5FC1D4706515BE48BE79A1C871
sa-east-1
CB454452665937052981CA118417B7A162A25F54
ca-central-1
1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1
ap-southeast-1
F719C49FEA86549E159818880E392C1570C953B6
ap-southeast-2
0148872FA02F3A7D6B38AA88FA5397228B28E08B
eu-central-1
9884072430220E6253011B88F940E4F20F53D0CC
us-east-1
598ADECB9A3E6CC70AA53D64BD5EE4704300382A
us-east-2
750B948515281953BC6F3D717A1E1654ECBFA852
us-west-1
89BABC6D46502653516CC0BA38B14A2B7864D161
us-west-2
63966130761608209718C5045CFFB4856FB53976

You can define your variable like this:

thumbprint_list = ["3ACE16CA6BAE7B16AAE3707096D1DE7D29093AD8","59ADE3A3A6039BBA092E920FEE413466493F409C","3FB881BCACBD420928168C739B07EEF47555946B","7CA6BE9F14E20973CB2C58452DA9B1E2BEB7236B","CAB073498D7558FEC3B2C414C006ACBA30805431","2EAFC197C15CDEE5426BFD4D27D3321A685F3B78","B715DC079832DA5FC1D4706515BE48BE79A1C871","CB454452665937052981CA118417B7A162A25F54","1C8B5245E80A6B7A0E8BF5FFDAB032273D7D5DF1","F719C49FEA86549E159818880E392C1570C953B6","0148872FA02F3A7D6B38AA88FA5397228B28E08B","9884072430220E6253011B88F940E4F20F53D0CC","598ADECB9A3E6CC70AA53D64BD5EE4704300382A","750B948515281953BC6F3D717A1E1654ECBFA852","89BABC6D46502653516CC0BA38B14A2B7864D161","63966130761608209718C5045CFFB4856FB53976"]

command for get fingerprint :

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

@scardena
Copy link

@scardena scardena commented May 25, 2020

@romaryoricardo I haven't test this yet, but I recommend @mzupan (or @cippaciong for macos) workaround over yours. I am not sure why this happens, but the fingerprints generated using your command are not the same as the ones generated by @mzupan commands (I am going to throw the guess that multiple certificates are being spit out for each command, but only 1 is processed, in @mzupan case is the most recent one, and in your case is the oldest one, although i can be completely wrong). It doesn't mean your solutions shouldn't work, but if we inspect the certificate of the fingerprint that your command spits, we can see that this certificate is about to expire (check the validity, it expires september 2020):

echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout -text

whereas in the other, it expires on 2034.
More over, I ran @mzupan script for all regions, and the output fingerprint seems to be the same for absolutely all regions (as @willthames suggests), so in my case, we decided to simply hardcode the fingerprint in the terraform module, although we still have the workaround commented out for future reference. We know hardcoding stuff sucks, but the workarounds presented are not too portable.

@wojciesze
Copy link

@wojciesze wojciesze commented Jun 16, 2020

For those who are not happy with dirty tricks in shell script: get_thumbprint.sh (e.g: tac twice).
Here is a working python script: (tested with python3)

import socket
from OpenSSL import SSL
import certifi
import sys, json

hostname = f'oidc.eks.{sys.argv[1]}.amazonaws.com'
port = 443


context = SSL.Context(method=SSL.TLSv1_METHOD)
context.load_verify_locations(cafile=certifi.where())

conn = SSL.Connection(context, socket=socket.socket(socket.AF_INET, socket.SOCK_STREAM))
conn.settimeout(5)
conn.connect((hostname, port))
conn.setblocking(1)
conn.do_handshake()
conn.set_tlsext_host_name(hostname.encode())

thumbprint = conn.get_peer_cert_chain()[-1].digest("sha1")
obj = {"thumbprint": thumbprint.decode("utf-8").replace(":", "").lower() }
print(json.dumps(obj))
conn.close()

For alpine docker images it enough to install:

apk add py3-openssl
pip3 install certifi

For python:3 docker image:

pip3 install pyopenssl certifi

@guitarrapc
Copy link

@guitarrapc guitarrapc commented Jun 23, 2020

#10104 (comment) used to be show longer certificate, which expire 2034.

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "9e99a48a9960b14926bb7f3b02e22da2b0ab7280"}

However I've ran same script and it shows shorter certificate, it expires on Sep 2020. (And is match to certificate show on browser.)

endpoint: oidc.eks.ap-northeast-1.amazonaws.com

{"thumbprint": "b715dc079832da5fc1d4706515be48be79a1c871"}

Certificate Chain

, CN=oidc.eks.ap-northeast-1.amazonaws.com, B715DC079832DA5FC1D4706515BE48BE79A1C871, 08/09/2020 21:00:00
, CN=Amazon, OU=Server CA 1B, O=Amazon, C=US, 917E732D330F9A12404F73D8BEA36948B929DFFC, 19/10/2025 09:00:00
, CN=Amazon Root CA 1, O=Amazon, C=US, 06B25927C42A721631C1EFD9431E648FA62E1E39, 31/12/2037 10:00:00
, CN=Starfield Services Root Certificate Authority - G2, O="Starfield Technologies, Inc.", L=Scottsdale, S=Arizona, C=US, 9E99A48A9960B14926BB7F3B02E22DA2B0AB7280, 29/06/2034 02:39:16
Starfield Class 2 Certification Authority, OU=Starfield Class 2 Certification Authority, O="Starfield Technologies, Inc.", C=US, AD7E1C28B064EF8F6003402014C3D0E3370EB58A, 30/06/2034 02:39:16

image

@marcincuber
Copy link
Author

@marcincuber marcincuber commented Jun 23, 2020

@guitarrapc this is a correct value for your region. B715DC079832DA5FC1D4706515BE48BE79A1C871. The initial value is for a different region.
See previous comment #10104 (comment)

I feel like this is an issue that can now be closed. There are numerous solutions suggested that will solve problems with OIDC provider.

@nickatsegment
Copy link

@nickatsegment nickatsegment commented Jul 14, 2020

Oof, I just created a new cluster today and it has a new cert for us-west-2:

$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0f:a2:dc:f6:0e:ff:f7:41:2f:76:c8:3c:e2:71:0d:6f
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Amazon, OU=Server CA 1B, CN=Amazon
        Validity
            Not Before: Jul 10 00:00:00 2020 GMT
            Not After : Aug 10 12:00:00 2021 GMT
$ region_name=us-west-2; echo | openssl s_client -servername oidc.eks.${region_name}.amazonaws.com -connect oidc.eks.${region_name}.amazonaws.com:443 2>&- | sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' | openssl x509 -fingerprint -noout | sed 's/://g' | awk -F= '{print $2}'

7FF5BDFA36CD78827820278B4DABB5E1E97C1476

@tyrken
Copy link

@tyrken tyrken commented Jul 14, 2020

@nickatsegment I'm thinking this is as you've made the same mistake I made and are using the "leaf" server cert which gets updated owing to it's short expiry. Instead use the root Certificate Authority cert which expires in 2034 or similar.

Switch to use the commands above which have -showcerts in the openssl command line to show all certs in the chain, but beware most force you to use a line reordering utility to pick out the right cert, which changes depending on the OS you run: tac (Linux) or tail -r OS/X.

@justcompile
Copy link

@justcompile justcompile commented Jul 14, 2020

Taken from a number of sources but with explicit help from this article https://medium.com/@michael.kandelaars/did-your-eks-iam-service-account-roles-break-today-2ea50c869aee, this thread & of course Stackoverflow for bash-fu, I've got this script which grabs the root ca thumbprint.

Not the prettiest, but function over form right?

#!/usr/bin/env bash

echo | openssl s_client -servername oidc.eks.${1}.amazonaws.com -showcerts -connect oidc.eks.${1}.amazonaws.com:443 2>&- | awk '/-----BEGIN/{f="cert."(n++)} f{print>f} /-----END/{f=""}'
 
certificates=()

for c in cert.*; do
   certificates+=($(openssl x509 <$c -noout -fingerprint))
done
rm cert.* 

thumbprint=$(echo ${certificates[${#certificates[@]}-1]} | sed 's/://g' | awk -F= '{print tolower($2)}')
thumbprint_json="{\"thumbprint\": \"${thumbprint}\"}"
echo $thumbprint_json

./this_script.sh <aws-region-code>

@nickatsegment
Copy link

@nickatsegment nickatsegment commented Jul 14, 2020

@olfway
Copy link

@olfway olfway commented Jul 17, 2020

My external for this

external/thumbprint

#!/bin/bash

set -euo pipefail

HOST="oidc.eks.$1.amazonaws.com"

# https://github.com/terraform-providers/terraform-provider-aws/issues/10104

echo | openssl s_client -servername "$HOST" -showcerts -connect "$HOST:443" 2>&- \
  | sed -n '/-----BEGIN CERTIFICATE-----/h;//!H;$!d;x;s/\(.*-----END CERTIFICATE-----\).*/\1/p' \
  | openssl x509 -fingerprint -sha1 -noout \
  | tr '[:upper:]' '[:lower:]' \
  | sed 's/://g; s/.*=\(.*\)/{"thumbprint": "\1"}/'

Use it like this:

data "external" "thumbprint" {
  program = [
    "/bin/sh",
    "${path.module}/external/thumbprint",
    data.aws_region.current.name,
  ]
}

resource "aws_iam_openid_connect_provider" "this" {
  url = aws_eks_cluster.this.identity[0]["oidc"][0]["issuer"]
  client_id_list = [
    "sts.amazonaws.com",
  ]
  thumbprint_list = [
    data.external.thumbprint.result.thumbprint,
  ]
}

For eu-central-1 region it returns fingerprint for this cert:

Issuer: C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
Validity
    Not Before: Sep  2 00:00:00 2009 GMT
    Not After : Jun 28 17:39:16 2034 GMT
Subject: C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2

@ghost
Copy link

@ghost ghost commented Jul 23, 2020

I'm going to lock this issue because it has been closed for 30 days . This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@hashicorp hashicorp locked and limited conversation to collaborators Jul 23, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement service/iam
Projects
None yet