Cannot trigger S3 backup using OIDC provider and IRSA credentials

[11qu1d]

Overview

I am trying to setup cluster backups without having to directly use AWS credentials, instead utilizing IRSA and OIDC.

Not 100% sure if this is a bug but looks like it.

Environment

Please provide the following details:

• Platform: EKS
• Platform Version: 1.21
• PGO Image Tag: ubi8-5.0.5-0
• Postgres Version: 13
• Storage: gp3

Steps to Reproduce

1. Create IAM role in AWS and assign a trust relationship with the cluster's OIDC provider.
2. Add relevant annotations to PGO's service account
3. Add following cluster resource

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
  labels:
    argocd.argoproj.io/instance: my-cluster
  name: my-cluster
  namespace: my-cluster
spec:
  backups:
    pgbackrest:
      configuration:
        - secret:
            name: my-cluster-pgbackrest-secret
      global:
        repo1-path: /pgbackrest/my-cluster/repo1
        repo1-retention-full: '14'
        repo1-retention-full-type: time
        repo1-s3-key-type: web-id
      manual:
        options:
          - '--type=full'
        repoName: repo1
      repos:
        - name: repo1
          s3:
            bucket: my-cluster
            endpoint: s3.amazonaws.com
            region: us-east-1
          schedules:
            full: 0 0 * * *
            incremental: 0 */4 * * *
  instances:
    - dataVolumeClaimSpec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: gp3
      name: my-cluster
      replicas: 3
  metadata:
    annotations:
      eks.amazonaws.com/role-arn: 'arn:aws:iam::1234567890:role/my-cluster-role'
  monitoring:
    pgmonitor:
      exporter:
        image: ''
  postgresVersion: 13
  proxy:
    pgBouncer:
      replicas: 1
  users:
    - databases:
        - my-db1
      name: my-user1
    - databases:
        - my-db2
      name: my-user2

4. Wait for the scheduled backup or trigger a manual backup.

EXPECTED

Backup job is triggered.

ACTUAL

Job is not triggered. Seeing the following errors in the logs

Logs

time="2022-04-04T23:45:47Z" level=error msg="unable to create stanza" error="command terminated with exit code 29: ERROR: [029]: unable to find child 'AssumeRoleWithWebIdentityResult':0 in node 'ErrorResponse'\n" file="internal/controller/postgrescluster/pgbackrest.go:2578" func="postgrescluster.(*Reconciler).reconcileStanzaCreate" name=my-cluster namespace=my-cluster reconciler=pgBackRest reconciler group=postgres-operator.crunchydata.com reconciler kind=PostgresCluster version=5.0.5-0

Additional Information

I have followed AWS troubleshooting instructions from here: https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/ but all looks correct.

Thanks

[benoitm76]

Hello,

This error probably means you assume role policy document is not properly configured. Check specified namespace and service account inside the policy. To debug it you can launch a pod with the aws CLI with the service account configured and run for example:
aws s3 ls

You will normally get an assume role error.

[bkrugerp99]

Hello,

This error probably means you assume role policy document is not properly configured. Check specified namespace and service account inside the policy. To debug it you can launch a pod with the aws CLI with the service account configured and run for example: aws s3 ls

You will normally get an assume role error.

[Editing my comment here]

If I follow this exactly:
https://docs.aws.amazon.com/eks/latest/userguide/create-service-account-iam-policy-and-role.html

crunchy-operator does not like this default policy with a sub.

Works:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::REDACTED:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/REDACTED"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.us-west-2.amazonaws.com/id/REDACTED:aud": "sts.amazonaws.com",
        }
      }
    }
  ]
}

You can remove the sub, that'll allow it to work, but not necessarily recommended. After going back over it today, I was able to re-insert the sub and things did work. After wiping the s3 bucket when it was working, it does go back to not working.

[bkrugerp99]

apiVersion: postgres-operator.crunchydata.com/v1beta1
kind: PostgresCluster
metadata:
labels:
argocd.argoproj.io/instance: my-cluster
name: my-cluster
namespace: my-cluster
spec:
backups:
pgbackrest:
configuration:
- secret:
name: my-cluster-pgbackrest-secret
global:
repo1-path: /pgbackrest/my-cluster/repo1
repo1-retention-full: '14'
repo1-retention-full-type: time

    repo1-s3-key-type: web-id

^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This likely needs to go into the secret.

Choosing key: s3.conf
[global]
repo1-s3-key-type=web-id
repo1-s3-key=1
repo1-s3-key-secret=1

This seems to work for us otherwise, though we do run into some odds here and there with the above.

[tjmoore4]

@11qu1d Are you still seeing the described issue? v5.1.0 added support for AWS IAM roles with S3 with backups when PGO is deployed in EKS.

Please see the "Using an AWS-integrated identity provider and role" section of the documentation for more information on how to configure your cluster.

[andrewlecuyer]

@11qu1d as @tjmoore4 described, this feature should now be available in PGO v5.1. If you continue to have trouble, please let use know.

[javiermas]

Hey I'm not sure this should've been closed. We have version 5.1.2 deployed in our cluster and we have the same problem that @11qu1d describes. @tjmoore4's solution still works: remove the 'sub' line under Statement[0].Condition.StringEquals. If you're using AWS's web interface you can find that document in the role definition, under 'Trust relationships'. That line seems to be generated by default when following the documentation provided by CrunchyData and AWS.