skaffold with kaniko permission failure on aws ecr

[janwillies]

When using skaffold with kaniko on aws ecr I'm getting a 401 after the push to the repo is finished.

$ skaffold dev
Generating Tag for [123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar]...
Starting build...
Creating kaniko secret [aws-secret]...      
Creating docker config secret [docker-kaniko-secret]...
Building [123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar]...
Storing build context at /var/folders/2w/4fq5mg1x7nddytv75mjyn5640000gn/T/context-b9e0c8d55db2cd3903c0d32cb5094107.tar.gz
INFO[0000] Downloading base image node:9
[...]
2019/03/03 14:16:40 existing blob: sha256:da44c9274f481e1f9123a37bbe0cea77259a96afc4cb18cc23e632038558edae
2019/03/03 14:16:41 pushed blob sha256:d06d00f253a35f7297095702f9ab7eb50ba4cab9894260f598c6b59bcfbcb909
2019/03/03 14:16:41 pushed blob sha256:38bacccc92980dfa64ad1f5b980bfb580b4165436dc4eded2833f06a6994b888
2019/03/03 14:16:47 pushed blob sha256:77c43eaef55e6046178adb778ab6ee359f8d8a5dab62f4c2e67a005d8979a852
2019/03/03 14:16:48 123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar:2bd4bf1-dirty: digest: sha256:9b08c9f13a346847b6c9a2cb9ece4d74bfb86f57b1cdbe69c61b327db24a0eaa size: 1906
FATA[0102] exiting dev mode because first run failed: build failed: building [123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar]: kaniko build for [123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar]: getting digest: unsupported status code 401; body: Not Authorized

Expected behavior

skaffold should continue after pushing to the ecr repo

Actual behavior

after successfully pushing to ecr skaffold exits with an error

I have kaniko setup up to build with gitlab-ci on the same ec2 nodes and it works fine, the kaniko configuration also looks the same. I'm not sure what skaffold (or kaniko?) wants from the repo, even with ecr:* I'm getting the 401. Clearly the push works fine, because the image ends up in ecr.

Information

• Skaffold version: v0.24.0
• Operating system: os x
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta6
kind: Config
build:
  artifacts:
  - image: 123456889.dkr.ecr.eu-central-1.amazonaws.com/foo/bar
    context: .
    sync:
      '*.js': .
  kaniko:
    buildContext: 
      localDir: {}
    pullSecretName: aws-secret
    dockerConfig: 
      secretName: docker-kaniko-secret
    namespace: dev
    cache: {}
deploy:
  kubectl:
    manifests:
      - k8s-*

Steps to reproduce the behavior

1. create a policy which allows the ec2 instances to push to ecr:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:DescribeImages",
                "ecr:BatchGetImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:PutImage"
      ],
      "Resource": "arn:aws:ecr:eu-central-1:123456889:repository/*"
    }
  ]
}

I've also tried with ecr:*, no difference

2. create dummy aws credentials because skaffold needs one:

touch credentials
kubectl create secret generic aws-secret --from-file credentials

3. create credHelper configuration so that kaniko knows how to login to ecr:

kubectl create secret generic docker-kaniko-secret --from-file config.json

contents of config.json:

{ "credHelpers": { "123456889.dkr.ecr.eu-central-1.amazonaws.com": "ecr-login" }}

4. run skaffold dev

related #731

[azaiter]

I ran to this issue myself, kaniko pod is actually succeeding and pushing to ECR. But your local docker is not able to get image digest from ECR, can you check that your local dockerconfig has access to ECR?

[janwillies]

exactly that was the issue, thanks @azaiter. I don't even need to have docker running, it's just that an $(aws ecr get-login --no-include-email) was missing.

I guess it's needed for skaffold to get the latest hash?

[azaiter]

It's just what kaniko runner returns based on the image tag, it just happens to use local docker (library?) for that.
See: https://github.com/GoogleContainerTools/skaffold/blob/cff3918df8c165f2c4bb17b07f3ed50222f6a63c/pkg/skaffold/build/kaniko/run.go

[janwillies]

Thx. Apparantly it doesn't even need local docker, just the registry login details