Failures with ECR registry

[vfarcic]

Summary

When running an EKS or AKS cluster with tekton, pushing to container registry fails. The same works with GKE which uses container registry in the cluster. I did not observe the same problem with static Jenkins. The setup is "vanilla" new cluster created with jx create cluster.

Steps to reproduce the behavior

jx create cluster eks -n jx-rocks \
    -r $AWS_DEFAULT_REGION \
    --node-type t2.medium \
    --nodes 3 \
    --nodes-min 3 \
    --nodes-max 6 \
    --default-admin-password=admin \
    --default-environment-prefix tekton \
    --git-provider-kind github \
    --namespace cd \
    --no-tiller \
    --prow \
    --tekton \
    -b

jx create quickstart -l go -p jx-go -b

jx get build logs

...
error pushing image: failed to push to destination 036548781187.dkr.ecr.us-west-2.amazonaws.com/vfarcic/jx-go:0.0.2: unsupported status code 401; body:
{"component":"entrypoint","error":"wrapped process failed: exit status 1","level":"error","msg":"Error executing test process","time":"2019-04-09T19:35:47Z"}
...

Expected behavior

Pushing to the registry should work.

Actual behavior

Pushing to the registry fails.

Jx version

The output of jx version is:

NAME             VERSION
jx               1.3.1096
git              git version 2.20.1 (Apple Git-117)
Operating System Mac OS X 10.14.4 build 18E226

Jenkins type

• Classic Jenkins
• Serverless Jenkins

Kubernetes cluster

Tested in AKS and EKS.

Operating system / Environment

macOS

[vfarcic]

The problem happens both in AKS and EKS and it is related to container registry services. If I switch to in-cluster Docker Registry by adding echo "docker-registry: enabled: truetomyvalues.yaml`, everything works correctly.

[jstrachan]

looks like we need to figure out the IAM roles so that kaniko can post to ECR

[jstrachan]

ah - we need to create a custom kaniko docker image with the ECR + ACR docker credential helper binaries installed then it should just work I hope... https://github.com/jenkins-x/jenkins-x-builders-base/blob/master/builder-base/Dockerfile.common#L68-L81

[rawlingsj]

the kaniko executor already has the ECR credentials helper https://github.com/GoogleContainerTools/kaniko/blob/master/deploy/Dockerfile#L23-L25 and https://github.com/GoogleContainerTools/kaniko/blob/master/deploy/Dockerfile#L33 we can add the ACR one too but I wonder if there is another reason this is failing?

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[deanesmith]

@vfarcic this issue should now be resolved. Can you confirm? @dgozalo confirmed with recent efforts for Kaniko support on EKS should be good now.

[mb-lyrical]

This is still a problem on AKS, I'm getting a similar error when running a simple PHP quickstart.

My system is set to use the acr-linux credentials but I get the error:

Showing logs for build xxx/demo-php-helloworld/master #3 stage from-build-pack and container step-build-container-build
INFO[0000] Downloading base image php:7.0-apache
2020/03/05 19:42:46 No matching credentials were found, falling back on anonymous
INFO[0002] Error while retrieving image from cache: getting file info: stat /workspace/sha256:b4e5b78afa2504589f3a9868535e2b099418c9de14711574b09691fc3e0f0054: no such file or directory
INFO[0002] Downloading base image php:7.0-apache
2020/03/05 19:42:49 No matching credentials were found, falling back on anonymous
INFO[0004] Using files from context: [/workspace/source/src]
INFO[0004] cmd: EXPOSE
INFO[0004] Adding exposed port: 80/tcp
INFO[0004] Skipping unpacking as no commands require it.
INFO[0004] Taking snapshot of full filesystem...
INFO[0004] Using files from context: [/workspace/source/src]
INFO[0004] COPY src/ /var/www/html
INFO[0004] Taking snapshot of files...
INFO[0004] EXPOSE 80
INFO[0004] cmd: EXPOSE
INFO[0004] Adding exposed port: 80/tcp
INFO[0004] No files changed in this command, skipping snapshotting.
error pushing image: failed to push to destination xxx.azurecr.io/xxx/demo-php-helloworld:0.0.4: unsupported status code 401; body:

I have forced docker credentials into the JX namespace as well by running:

jx create docker auth --host ...(args)

And now I can get remote docker images but not push.

[mb-lyrical]

so when I add the docker auth and set the repository I get this on a build:

Build logs for xxx/test-php-helloworld/master #3

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-credential-initializer-l7gxs
{"level":"warn","ts":1583625960.7274427,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-jx-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1583625960.7282836,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-working-dir-initializer-68kj5
{"level":"warn","ts":1583625961.6639283,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1583625961.6686976,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-place-tools

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-git-source-meta-xxx-test-php-hello-lrr7c-ww2wt
{"level":"warn","ts":1583625967.7366695,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-jx-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1583625970.6668763,"logger":"fallback-logger","caller":"git/git.go:103","msg":"Successfully cloned https://github.com/xxx/test-php-helloworld.git @ 39f59984da5882469b78269ef6d0320ee80f60cd in path /workspace/source"}

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-git-merge
Using SHAs from PULL_REFS=master:39f59984da5882469b78269ef6d0320ee80f60cd
WARNING: no SHAs to merge, falling back to initial cloned commit

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-merge-pull-refs
SKIP merge-pull-refs: Nothing to merge

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-create-effective-pipeline
Effective pipeline written to jenkins-x-effective.yml

Showing logs for build xxx/test-php-helloworld/master #3 stage meta-pipeline and container step-create-tekton-crds
running command: jx step next-version --use-git-tag-only --tag
created new version: 0.0.3 and written to file: ./VERSION
Updating chart version in charts/test-php-helloworld/Chart.yaml to 0.0.3
Updating repository in charts/test-php-helloworld/values.yaml to https://whistledev.azurecr.io/xxx/test-php-helloworld
Updating tag in charts/test-php-helloworld/values.yaml to 0.0.3
Tag v0.0.3 created and pushed to remote origin

WARNING: failed to find secret kaniko-secret in namespace jx: secrets "kaniko-secret" not found
PipelineActivity for xxx-test-php-helloworld-master-3
Applying changes
upserted PipelineResource xxx-test-php-helloworld-f7f6p for the git repository https://github.com/xxx/test-php-helloworld.git
upserted Task xxx-test-php-helloworld-f7f6p-from-build-pack-3
upserted Pipeline xxx-test-php-helloworld-f7f6p-3
created PipelineRun xxx-test-php-helloworld-f7f6p-3
created PipelineStructure xxx-test-php-helloworld-f7f6p-3

waiting for stage from-build-pack : container step-credential-initializer-d5m9w to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-credential-initializer-d5m9w
{"level":"warn","ts":1583626003.6650426,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-jx-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1583626003.6654384,"logger":"fallback-logger","caller":"creds-init/main.go:40","msg":"Credentials initialized."}

waiting for stage from-build-pack : container step-working-dir-initializer-f67wg to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-working-dir-initializer-f67wg
{"level":"warn","ts":1583626005.4145267,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: open /var/run/ko/HEAD: no such file or directory"}
{"level":"info","ts":1583626005.4596276,"logger":"fallback-logger","caller":"bash/main.go:64","msg":"Successfully executed command \"sh -c mkdir -p /workspace/source\"; output "}

waiting for stage from-build-pack : container step-place-tools to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-place-tools

waiting for stage from-build-pack : container step-git-source-xxx-test-php-helloworld-f7f6p-t7wbj to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-git-source-xxx-test-php-helloworld-f7f6p-t7wbj
{"level":"warn","ts":1583626023.6685417,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"ref: refs/heads/0.8.0-jx-support-backwards-incompats\" is not a valid GitHub commit ID"}
{"level":"info","ts":1583626026.9545398,"logger":"fallback-logger","caller":"git/git.go:103","msg":"Successfully cloned https://github.com/xxx/test-php-helloworld.git @ v0.0.3 in path /workspace/source"}

waiting for stage from-build-pack : container step-git-merge to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-git-merge
Using SHAs from PULL_REFS=master:39f59984da5882469b78269ef6d0320ee80f60cd
WARNING: no SHAs to merge, falling back to initial cloned commit

waiting for stage from-build-pack : container step-setup-jx-git-credentials to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-setup-jx-git-credentials
Generated Git credentials file /workspace/xdg_config/git/credentials

waiting for stage from-build-pack : container step-build-container-build to start...


Showing logs for build xxx/test-php-helloworld/master #3 stage from-build-pack and container step-build-container-build
INFO[0000] Downloading base image php:7.0-apache
INFO[0000] Error while retrieving image from cache: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-acr-linux: exec: "docker-credential-acr-linux": executable file not found in $PATH; output:
INFO[0000] Downloading base image php:7.0-apache
error building image: Get https://auth.docker.io/token?scope=repository%3Alibrary%2Fphp%3Apull&service=registry.docker.io: invoking docker-credential-acr-linux: exec: "docker-credential-acr-linux": executable file not found in $PATH; output:

Pipeline failed on stage 'from-build-pack' : container 'step-build-container-build'. The execution of the pipeline has stopped.

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.