connaisseur+cosign on eks: add support for aws credentials

[hsuchan]

k8s version: 1.20
flavor: eks

Goal:

Use connaisseur+cosign on eks to sign/validate images located in private ecr registry.

Problem statement:

At the moment connaisseur provides 2 methods to auth against a private registry:
(https://github.com/sse-secure-systems/connaisseur/blob/master/helm/values.yaml#L82-L87)

1. A secret containing the registry's credentials (i.e. config.json)
2. A username/password

Since docker doesn't natively support ecr auth, in order to auth to ecr, one needs to use an aws helper:
(https://docs.aws.amazon.com/AmazonECR/latest/userguide/registry_auth.html)

$ TOKEN=$(aws ecr get-login-password --region region)

This creates a temporary token valid for 12 hours, which you can then use to authenticate to ecr:

$ echo $TOKEN | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com

Creating a secret with the following information would unfortunately expire within 12h.

{
  "auths": {
    "aws_account_id.dkr.ecr.region.amazonaws.com": {
      "username": "AWS",
      "password": "<TOKEN>"
    }
  }
}

The solution I had to come up with as reported by @diegonicacio in #352 was to create a cronjob, whose job is to rotate the ecr credentials inside the secret every x hours.

While this works, it's a fragile solution, and it would be a more elegant solution if connaisseur was able to natively support AWS credentials:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

[xopham]

@hsuchan like the idea to avoid extra steps here. It is very unfortunate that apparently every cloud provider rolls their own docker credential helper which would mean to continuously extend Connaisseur by external code/SDKs. There seems to be a credential helper helper available that might resolve part of the technicalities and an open docker issue.
Are you aware of any other open source project that has found a pragmatic solution? It seems to be a common issue.

[ZacHaque]

I am tackling the situation the same way as @hsuchan doing, Running a job to run for the very first time then cronJob run job every x hours to refresh the token + replace all pods on the deployment. I also tired if irsa could help me out by giving connaisseur serviceaccount ecr permission but no luck. I will look forward if anyone have a better solution.

[marckn0x]

I've implemented passing --k8s-keychain in #551 , which will probably work for you @hsuchan . I explain in the documentation my PR adds: https://github.com/sse-secure-systems/connaisseur/pull/551/files#diff-b3134ab5d416e3bda59da012fc24c95d4815ed280d2bbc136d4a6bb98bd7409bR143

[xopham]

there has been many good contributions now to improve support for EKS. Maybe we should prepare a blog post on how to best use Connaisseur with AWS EKS to spread the knowledge or eventually add full examples to the docs somewhere

[xopham]

@hsuchan the feature has been released with v2.5.0. Would be excited to see whether that resolves the issue: https://sse-secure-systems.github.io/connaisseur/v2.5.0/validators/sigstore_cosign/#k8s_keychain

[hsuchan]

@sf-jmarcou and @marckn0x I really appreciate the effort to add EKS credentials support to connaisseur.

@xopham I was able to test both:

• the extra configuration in helm chart solution (i.e. refresher sidecar)
• the "--k8s-keychain" flag feature solution

both solutions work great so far.

[xopham]

wow...awesome! Super happy that support of AWS has improved so much thanks to your efforts 🚀
Closing the issue, as it seems we have a suitable solution here. Feel free to re-open if anything comes up.