EOF on AWS ECR API POST to check push permissions

[MaesterZ]

Actual behavior
Pushing a tag different than latest fails on AWS ECR and works fine GitLab container registry (using a single destination arg)

Expected behavior
You can push Docker images to AWS ECR (and GitLab container registry)

To Reproduce
Using GitLab to build a Docker image with Kaniko v1.7.0-debug following this guide https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
Using docker image sha256:f39fe1b68ae07abd6ce979135e84ae10495c961b5d90863c14295a16c3dd91b7 for gcr.io/kaniko-project/executor:v1.7.0-debug with digest gcr.io/kaniko-project/executor@sha256:88dacc7ea3f5c04709eae96776693c717869405364b19d6e78850fe54c63c6a2 ...

Error:
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<path_to_repo>": Post "https://<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/v2/tezos-ci/blobs/uploads/": EOF

Docker JSON configuration for GitLab container registry and Amazon ECR:

mkdir -pv /kaniko/.docker
CI_REGISTRY_AUTH=$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')
echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"${CI_REGISTRY_AUTH}\"}},\"credHelpers\":{\"$AWS_ECR\":\"ecr-login\"}}" > /kaniko/.docker/config.json

Kaniko command:

/kaniko/executor
--context /builds/<path_to_project>
--dockerfile /builds/<path_to_project>/Dockerfile
--destination registry.gitlab.com/<path_to_repo>:test
--destination <aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<path_to_repo>:test

Simple Dockerfile:

FROM alpine:3.15.0

Description	Yes/No
Please check if this a new feature you are proposing	No
Please check if the build works in docker but not in kaniko	Yes
Please check if this error is seen when you use --cache flag	Yes
Please check if your dockerfile is a multistage dockerfile	No

Not sure if it is similar to #1792 or even #1209

[imjasonh]

Kaniko v1.7.0 had some auth bugs, and was reverted so that :latest points at :v1.6.0.

Can you either try this with v1.6.0, or even better, try a recent commit-tagged build to help us check whether this is still a bug at head, in advance of an upcoming v1.8.0 release (#1871)

[MaesterZ]

Tried out 76624697df879f7c3e3348f22b8c986071af4835-debug and v1.6.0-debug, no luck, I got the same error :(

WARN[0000] Failed to setup Google env authenticator, ignoring: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. 
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/<path_to_repo>": Post "https://<aws_account_id>.dkr.ecr.<aws_region>.amazonaws.com/v2/tezos-ci/blobs/uploads/": EOF

[MaesterZ]

I am able to reproduce with Docker, I think it might be an AWS ECR issue, still digging ...

[MaesterZ]

@imjasonh found my problem, it was AWS IAM permissions issues, my pipeline was using the EC2 instance IAM role instead of credentials in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Reading the Docker Daemon logs and AWS ECR credential helper logs in ~/.ecr/log/ecr-login.log helped a lot.

Related to awslabs/amazon-ecr-credential-helper#183

[Jasper-Ben]

@imjasonh found my problem, it was AWS IAM permissions issues, my pipeline was using the EC2 instance IAM role instead of credentials in environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Reading the Docker Daemon logs and AWS ECR credential helper logs in ~/.ecr/log/ecr-login.log helped a lot.

Related to awslabs/amazon-ecr-credential-helper#183

Running into the same issue: #1455 (comment)

does this mean that there currently is no way to get this working with IRSA?

[AuboIoT]

For me, what make it work is this:
ECR repository name === container image name.

[cartermckinnon]

ECR repository name === container image name.

This was my issue as well, you cannot push foo/bar:tag to an ECR repository named foo. It must be named foo/bar.