basic-auth credentials not working with Kaniko task for registry push, ServiceAccount secrets

[doctorpangloss]

Is there something I have to do to make a Pipeline correctly populate the kaniko task with the stuff it needs?

Expected Behavior

I should be able to specify basic-auth credentials in a Secret, reference the Secret in a ServiceAccount, then build and push an image.

Actual Behavior

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "[...]...)": POST .../blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: projects/..., action: push: unauthorized to access repository: projects/..., action: push

docker login myserver.com and pushing from local works fine with these credentials.

Mounting a dockerconfig manually using a workspace works fine with these credentials.

Steps to Reproduce the Problem

Create an Pipeline, Secret and ServiceAccount, then submit a PipelineRun that use a kaniko task for a well-known thing.

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.6+k0s
WARNING: version difference between client (1.28) and server (1.30) exceeds the supported minor version skew of +/-1

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.58.0

Manifests

Working:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  annotations:
    organization.maxAllowedCompleted: "1"
    organization.maxAllowedRunning: "1"
  name: build-public-ip-checker
  namespace: organization-namespace
spec:
  description: |
    This pipeline clones the organization repository, builds a Docker image
    for the public IP checker utility and pushes it to Registry registry
  params:
    - name: repo-url
      type: string
      description: Git repository URL
      default: https://github.com/Organization/organization-repository.git
    - name: image-reference
      type: string
      description: Docker image reference
      default: registry.organization.com/projects/public-ip-checker
    - name: dockerfile-path
      type: string
      description: Path to the Dockerfile
      default: bootstrap/organization-directory/Dockerfile
    - name: context-path
      type: string
      description: Path to the build context
      default: bootstrap/organization-directory
    - name: toml-path
      type: string
      description: Path to the pyproject.toml file
      default: bootstrap/organization-directory/pyproject.toml
  workspaces:
    - name: shared-data
      description: Workspace containing the cloned git repository and build context
    - name: dockerconfig
      description: Docker credentials
  tasks:
    - name: fetch-source
      taskRef:
        name: git-clone
        kind: ClusterTask
      workspaces:
        - name: output
          workspace: shared-data
      params:
        - name: url
          value: $(params.repo-url)
        - name: deleteExisting
          value: "true"
    - name: extract-version
      runAfter: ["fetch-source"]
      taskRef:
        name: python-script
        kind: ClusterTask
      params:
        - name: script
          value: |
            #!/usr/bin/env python3
            import tomli
            import os
            toml_path = "$(params.toml-path)"
            with open(toml_path, "rb") as f:
                pyproject = tomli.load(f)
            
            version = str(pyproject["project"]["version"]).strip()
            print(version, end="")
        - name: packages
          value: "tomli"
      workspaces:
        - name: source
          workspace: shared-data
    - name: build-push
      runAfter: ["extract-version"]
      taskRef:
        name: kaniko
        kind: ClusterTask
      params:
        - name: IMAGE
          value: $(params.image-reference):$(tasks.extract-version.results.stdout)
        - name: DOCKERFILE
          value: $(params.dockerfile-path)
        - name: CONTEXT
          value: $(params.context-path)
        - name: EXTRA_ARGS
          value:
            - --destination=$(params.image-reference):latest
      workspaces:
        - name: source
          workspace: shared-data
        - name: dockerconfig
          workspace: dockerconfig
---
apiVersion: v1
kind: Secret
metadata:
  name: public-ip-checker-github-ssh-auth
  namespace: xxx
  annotations:
    tekton.dev/git-0: https://xxx
type: kubernetes.io/basic-auth
stringData:
  username: "xxx"
  password: "xxx"
---
apiVersion: v1
kind: Secret
metadata:
  name: public-ip-checker-harbor-basic-auth
  namespace: xxx
  annotations:
    tekton.dev/docker-0: https://xxx.com
type: kubernetes.io/basic-auth
stringData:
  username: "xxx"
  password: "xxx"
---
apiVersion: v1
kind: Secret
metadata:
  name: public-ip-checker-harbor-docker-auth
  namespace: xxx
  annotations:
    tekton.dev/docker-0: xxx.com
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "xxx.com": {
          "auth": "xxx"
        }
      }
    }
type: kubernetes.io/dockerconfigjson

---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  annotations:
    organization.maxAllowedCompleted: '1'
    organization.maxAllowedRunning: '1'
  generateName: public-ip-checker-build-xxxxx-r-
  labels:
    dashboard.tekton.dev/rerunOf: public-ip-checker-build-xxxxx-r-xxxxx
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  namespace: organization-namespace
spec:
  params:
    - name: repo-url
      value: https://github.com/Organization/organization-repository.git
    - name: image-reference
      value: registry.organization.com/projects/public-ip-checker
    - name: dockerfile-path
      value: bootstrap/organization-directory/Dockerfile
    - name: context-path
      value: bootstrap/organization-directory
  pipelineRef:
    name: build-public-ip-checker
  taskRunTemplate:
    podTemplate:
      nodeSelector:
        kubernetes.io/os: linux
    serviceAccountName: ip-checker-pipeline-bot
  timeouts:
    pipeline: 1h0m0s
  workspaces:
    - name: dockerconfig
      secret:
        items:
          - key: .dockerconfigjson
            path: config.json
        secretName: public-ip-checker-registry-docker-auth
    - name: shared-data
      volumeClaimTemplate:
        metadata:
          creationTimestamp: null
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
        status: {}
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ip-checker-pipeline-bot
  namespace: xxx
secrets:
  - name: public-ip-checker-github-ssh-auth
  - name: public-ip-checker-harbor-docker-auth

The pipeline which does not use the dockerconfig and instead relies on functionality from the docs for mounting docker basic auth does not work. Corresponding service account:

...
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ip-checker-pipeline-bot
  namespace: xxx
secrets:
  - name: public-ip-checker-github-ssh-auth
  - name: public-ip-checker-harbor-basic-auth

[AlanGreene]

The pipeline which does not use the dockerconfig and instead relies on functionality from the docs for mounting docker basic auth does not work.

Can you share a link for the docs you're referring to here?

Tekton Pipeline version:
…
v0.58.0

It's also worth pointing out that the version of Tekton Pipelines you're running was EOL a year ago.

the kaniko task

Can you confirm specifically which Task you're referring to? Is it the one from the Tekton Catalog or from some other source? Which version of the Task?

[doctorpangloss]

Thanks for taking a look at this.

Can you share a link for the docs you're referring to here?
https://tekton.dev/docs/pipelines/auth/#configuring-basic-auth-authentication-for-docker

Also related to: #7453 - the user here tries a variety of methods simultaneously before discovering that there's no "automatic" credentials mounting process for docker.

I think the expected behavior is that once a service account has an associated secrets key, tekton will set up directories in the right canonical places (such as the config.json for docker) with the right data for every container in every task's pod using that service account. This is kind of what's implied by the docs I linked, but of course, I know better, I know that tekton doesn't do that. It's hard to tell from the documentation is all.

Why this works with the ssh secret and not the docker secret - I don't know!

Can you confirm specifically which Task you're referring to? Is it the one from the Tekton Catalog or from some other source? Which version of the Task?

Yes, kaniko from the catalog. we use 0.5 but it is not materially different from 0.7