aws rds generate-db-auth-token creates invalid token when executed inside ECS container

[ejoebstl]

I've created a small service which uses IAM to connect to a postgres database, according to the documentation.

export PGPASSWORD=$(aws rds generate-db-auth-token --hostname ${POSTGRES_HOST} --port ${POSTGRES_PORT} --username ${POSTGRES_USER} --region ${REGION})
psql -h "${POSTGRES_HOST}" -p "${POSTGRES_PORT}" -U ${POSTGRES_USER} "dbname=${POSTGRES_DATABASE}  sslmode=verify-full sslrootcert=rds_root.pem"

This works fine when I execute it on the local machine using the same policy.

However, when I execute inside a container deployed to ECS, authentication fails with the returned token. The error is: psql: FATAL: PAM authentication failed for user "backup_user".

Also, the token returned by the command is significantly longer when invoked on ECS. More specifically, it contains an additional very large parameter called X-Amz-Security-Token=.

I have verified that all environment variables are equal. When the extra parameter is removed, the created tokens are equal except timestamps and signatures.

When the token from ECS is copied to a local machine (where the commands work), the authentication also fails.

[justnance]

@ejoebstl - Thank you for reaching out. Based on the information provided so far it appears the IAM role and related policies on the ECS containers may not have sufficient permission to connect to the RDS database. Please review the online documentation below and confirm the container has the correct IAM permissions. Just wanted to add:

Thanks.

ECS IAM Policies
Amazon ECS Task Role

[ejoebstl]

Hello,

yes, while I initially assumed that this is a cli issue, it seems like a permission issue. I'm currently investigating together with the ecs agent team here.

I'm closing this issue.

Thanks!

[Matv1989]

Hi @ejoebstl ,
I've spent about a week on this problem with AWS support and they confirm that there is an AWS issue. In my case I was not able to create from EC2, but I hope my forum thread would be helpful to you.
https://forums.aws.amazon.com/thread.jspa?threadID=291106
Basically, the thing is that role attached to EC2 generates incorrect token until role assumed explicitlycfrom either cli or sdk, and request to RDS (aws rds generate-db-auth-token) is made with temporary credentials generated by this explicit operation.

[ejoebstl]

Thank you - we came to the same conclusion from the other issue.

[ejoebstl]

No, the discussion continued in the referenced issue. I will ask to re-open it.

[priyeshn]

Is there a solution for this issue ? Im facing same problem when connecting to RDS from ECS/Fargate