Unable to push to cross-account AWS ECR registry

[davidski]

Behaviour

Reference: https://github.com/davidski/test-docker/runs/1552032319?check_suite_focus=true

Pushes to AWS ECR work when the authenticating user is in the same AWS account as the repository, but do not work when the repository is in a different account (but the calling account has permissions).

I realize there is a containerd debug approach recommended, but I'm unsure how to apply that procedure to an AWS ECR location with its particular auth dance of AWS credentials for a docker login. If there's guidance on how to do that, I'm happy to apply it and confirm where the problem lies.

Steps to reproduce this issue

1. Setup a static user in ACCOUNT A with full permissions to ECR.
2. Setup a repository in ACCOUNT B, giving ACCOUNT A full permissions at the account ID (root) level.
3. Push process fails with a 401.

I have confirmed that using these static credentials (from ACCOUNT A) I can push to ACCOUNT B repos from a local Docker (MacOS) installation, verifying that permissions are correct.

Expected behaviour

Push should work, just as it does if the repository is in ACCOUNT A.

Actual behaviour

Push fails with a 401.

Configuration

• https://github.com/davidski/test-docker
• https://github.com/davidski/test-docker/runs/1552032319?check_suite_focus=true

name: Docker Image

on:
  push:
    branches:
      - master
      - main

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@v2
    - name: Docker meta
      id: docker_meta
      uses: crazy-max/ghaction-docker-meta@v1
      with:
        images: ${{ secrets.AWS_ECR_REPOSITORY }}/docker-image
        tag-sha: true
    - name: Set up QEMU
      uses: docker/setup-qemu-action@v1
    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1
      with:
        buildkitd-flages: --debug
                -
      name: Set up containerd
      uses: crazy-max/ghaction-setup-containerd@v1
    - name: Login to AWS Elastic Container Registry
      uses: docker/login-action@v1
      with:
        registry: ${{ secrets.AWS_ECR_REPOSITORY }}
        username: ${{ secrets.AWS_ACCESS_KEY_ID }}
        password: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
    - name: Build and push
      id: docker_build
      uses: docker/build-push-action@v2
      with:
        push: ${{ github.event_name != 'pull_request' }}
        tags: ${{ steps.docker_meta.outputs.tags }}
        labels: ${{ steps.docker_meta.outputs.labels }}
    - name: Dump context
      if: always()
      uses: crazy-max/ghaction-dump-context@v1

Logs

logs_2.zip

[davidski]

Update, this turns out to be a problem/difference in the docker-login action. When running on ubuntu-latest, that action calls AWS CLI v1 and fails. Switching to ubuntu-20.04, which as AWS CLI v2, that action calls get-login-password and the build works without issue. I'll leave this open for now as I'm not sure if that's known behavior or if it's worth addressing (or at least documenting).

[crazy-max]

@jtoberon Any idea?

@davidski Can you try with the following steps:

      -
        name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: <region>
      -
        name: Login to ECR
        uses: docker/login-action@v1
        with:
          registry: ${{ secrets.AWS_ECR_REPOSITORY }}

Replace <region> with its respective value.

Also is ${{ secrets.AWS_ECR_REPOSITORY }} in the following form <aws-account-number>.dkr.ecr.<region>.amazonaws.com?

[jtoberon]

Just pinged the ECR team.

[davidski]

Hi, @crazy-max - I've tested out adding the explicit configure-aws-credentials step (and removing the creds from the build and push step). Under AWS CLI v2 (found on ubuntu-20.04), the push still succeeds and under AWS CLI v1 (found on ubuntu-latest) the push still fails.

Fail run (CLI v1): https://github.com/davidski/test-docker/runs/1564906561?check_suite_focus=true#step:8:65
Success run (CLI v2): https://github.com/davidski/test-docker/runs/1564931964?check_suite_focus=true

I know the ECR login commands got a big rework between v1 and v2, so something seems to be not functioning on the older API.

Oh, and yes, the AWS_ECR_REPOSITORY secret is a full <account>.dkr.ecr.<region>.amaazonaws.com string. Using separate variables for the region and account ID would probably be more flexible and clever. #223

Thanks for looking at this!

[michaelb990]

Hi @davidski!

It seems like this line is the issue in the docker/login-action command. get-login without a registry-id passed in will log you into the default registry for the user. So, if I'm reading this correctly, your action (when using aws-cli-v1) is logging into your default registry, in this case ACCOUNT A, instead of the registry that you're trying to login to in ACCOUNT B.

We can look into the best way to fix the issue with the docker/login-action, but in the meantime, you may want to try using the amazon-ecr-login action which shouldn't have this issue.

[michaelb990]

Filed docker/login-action#39 for this. I was able to reproduce. @crazy-max let me know if we can help with this.

[crazy-max]

Hi @michaelb990! Thanks for your feedback! I will take a look and come back to you shortely.