Why pipeline sometimes executes fast but sometimes slow?

[fighterhit]

I found that in my environment, the pipeline sometimes executes fast but sometimes slow. After several experiments and analysis, I found that all the initContainers are executed very quickly, but the last container will get stuck when executing my script and there is no output, so I want to debug the entrypoint program by importing runtime/pprof to analyze it which step is the most time-consuming when executing the script. So I need to remake the image, but I didn't find where the entrypoint Dockerfile is, or is there any other better way to solve this problem?

Here is my Task:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: dod-test
  labels:
    app.kubernetes.io/version: "0.1"
  annotations:
    tekton.dev/pipelines.minVersion: "0.17.0"
    tekton.dev/categories: Image Build
    tekton.dev/tags: image-build
    tekton.dev/displayName: "Build and upload container image using Docker outside Docker"
    tekton.dev/platforms: "linux/amd64"
spec:
  description: >-
    This Task builds source into a container image using Docker outside Docker.

  resources:
    outputs:
      - name: builtImage
        type: image
        description: The name of where to store the app image.
        optional: true

  params:
  # These may be overridden, but provide sensible defaults.
  - name: directory
    type: string
    description: The directory containing the build context.
    default: ./
  - name: dockerfileName
    type: string
    description: The name of the Dockerfile
    default: Dockerfile
  - name: image
    description: Name (reference) of the image to build.
  - name: proxy
    description: Proxy for image building
    default: "http://proxy-inner.dl.xxxx:8787"
  workspaces:
  - name: source
    description: Holds the context and docker file

  stepTemplate:
    env:
      - name: DOCKER_API_VERSION
        value: "1.38"
  steps:
  - name: docker-build-push
    image: xxxx/agent-lcap-hub/cloud-builders/docker
    imagePullPolicy: IfNotPresent
    workingDir: "$(workspaces.source.path)/$(params.directory)"
    script: |
      #!/usr/bin/env bash
      trap 'date' DEBUG
      set -euvx

      ssh-keyscan -p 48233 dev-k8s.xxxx >> ~/.ssh/known_hosts
      ssh-keyscan -p 2222 gitlab.xxxx >> ~/.ssh/known_hosts
      
      GIT_TRACE=2 git clone ssh://git@dev-k8s.xxxx:48233/sandbox-002/backend.git $(workspaces.source.path)/$(params.directory)

      docker build --tag $(params.image) --file $(params.dockerfileName) .
      docker --config $(workspaces.source.path)/.docker/ push $(params.image)
    volumeMounts:
      - name: docker-socket
        mountPath: /var/run/docker.sock
      - name: docker-secret    
        mountPath: $(workspaces.source.path)/.docker/

  # As an implementation detail, this Task mounts the host's daemon socket.
  volumes:
    - name: docker-socket
      hostPath:
        path: /var/run/docker.sock
        type: Socket

[pritidesai]

hey @fighterhit the images are built using ko.

You can take a look at how the images are built in our release task:

pipeline/tekton/publish.yaml

Lines 12 to 14 in 6cb0f4c

	- name: images
	description: List of cmd/* paths to be published as images
	default: "controller webhook entrypoint nop kubeconfigwriter git-init imagedigestexporter pullrequest-init workingdirinit"

Reference on how to build an image using ko: https://github.com/google/ko#build-an-image

I generally apply my changes to the source under cmd and build an image locally using ko.

[fighterhit]

Thanks for your kind help @pritidesai , I have solved it according to the ko document.