pipelines-in-pipelines causes pipelines-controller to segfault

[jwitrick]

Expected Behavior

When using pipelines-in-pipelines (I know its an alpha feature) i expect it be able to start the 2nd pipeline as expected.

Actual Behavior

When I create a pipelinerun (through the tekton dashboad) the pipelines-controller pod has a segfault and crashes. The logs are:

{"level":"info","ts":1709037401.0109792,"logger":"tekton-pipelines-controller.event-broadcaster","caller":"record/event.go:285","msg":"Event(v1.ObjectReference{Kind:\"PipelineRun\", Namespace:\"justin\", Name:\"manual1\", UID:\"87f68829-58c1-41fc-bed8-280959e84c22\", APIVersion:\"tekton.dev/v1\", ResourceVersion:\"4909350781\", FieldPath:\"\"}): type: 'Normal' reason: 'Started' ","commit":"9be03e2"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1ad3024]
goroutine 341 [running]:
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources.resolveTask({0x268dc30, 0x402a17de90}, 0x0, _, {{0x4000a799f0, 0xe}, {0x0, 0x0}, {0x0, 0x0}, ...})
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go:659 +0x2a4
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources.(*ResolvedPipelineTask).setTaskRunsAndResolvedTask(0x4010814d80, {0x268dc30, 0x402a17de90}, {_, _}, _, _, {{0x4000a799f0, 0xe}, {0x0, ...}, ...})
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go:615 +0x134
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources.ResolvePipelineTask({_, _}, {{{0x0, 0x0}, {0x0, 0x0}}, {{0x4000c676a0, 0x7}, {0x0, 0x0}, ...}, ...}, ...)
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/resources/pipelinerunresolution.go:587 +0x380
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun.(*Reconciler).resolvePipelineState(0x400015f0e0, {0x268dc30, 0x40193ad530}, {0x402b0c4c40, 0x2, 0x0?}, 0x400067a4b0, 0x402c3f6000, {0x3b603a0, 0x0, ...})
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/pipelinerun.go:365 +0x328
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun.(*Reconciler).reconcile(0x400015f0e0, {0x268dc30, 0x402966a870}, 0x402c3f6000, 0xffff80461200?, 0x401263dd50)
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/pipelinerun.go:610 +0x20a4
github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun.(*Reconciler).ReconcileKind(0x400015f0e0, {0x268dc30, 0x402966a6c0}, 0x402c3f6000)
    github.com/tektoncd/pipeline/pkg/reconciler/pipelinerun/pipelinerun.go:265 +0xd74
github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun.(*reconcilerImpl).Reconcile(0x400053e5a0, {0x268dc30, 0x402966a510}, {0x4000c67860, 0xe})
    github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun/reconciler.go:236 +0x438
knative.dev/pkg/controller.(*Impl).processNextWorkItem(0x40004c1aa0)
    knative.dev/pkg@v0.0.0-20231023150739-56bfe0dd9626/controller/controller.go:542 +0x37c
knative.dev/pkg/controller.(*Impl).RunContext.func3()
    knative.dev/pkg@v0.0.0-20231023150739-56bfe0dd9626/controller/controller.go:491 +0x5c
created by knative.dev/pkg/controller.(*Impl).RunContext
    knative.dev/pkg@v0.0.0-20231023150739-56bfe0dd9626/controller/controller.go:489 +0x2b0

The pipelines-controller pod will keep crashing every time it tries to start this pipeline run.

Steps to Reproduce the Problem

1. Using the following pipelines

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-security-scans
spec:
  tasks:
    - name: custom-vars-pip
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "GETTING HERE"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: custom-vars
      params:
        - name: test1
          value: $(params.test1)
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        params:
          - name: test1
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "Value of test1 is: "
              echo $(params.test1)
    - name: security-scans
      pipelineRef:
        name: pip-security-scans

2. Create a pipelinerun (through the tekton dashboard), the pipelinerun will show as either 'pending' or 'creating' but nothing will happen

Additional Info

• Kubernetes version:

  Output of kubectl version:

Client Version: v1.29.2
Server Version: v1.28.5-eks-5e0fdde

• Tekton Pipeline version:

  Output of tkn version or kubectl get pods -n tekton-pipelines -l app=tekton-pipelines-controller -o=jsonpath='{.items[0].metadata.labels.version}'

v0.56.1

Tekton Dashboard version: 0.43.1

[l-qing]

You can reproduce the issue by creating this pipelinerun resource, and it's not necessarily to install the tekton-dashboard.

---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: pip-panic
spec:
  pipelineRef:
    name: pip-clone-scan-notify
  params:
    - name: test1
      value: "test1"

[l-qing]

Using this method can avoid panic, but the occurrence of a panic is certainly not expected in the program.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: security-scans
      # pipelineRef:
      #   name: pip-security-scans
      taskRef:
        apiVersion: tekton.dev/v1
        kind: Pipeline
        name: pip-security-scans

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-security-scans
spec:
  tasks:
    - name: custom-vars-pip
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "GETTING HERE"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: custom-vars
      params:
        - name: test1
          value: $(params.test1)
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        params:
          - name: test1
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "Value of test1 is: "
              echo $(params.test1)
    - name: security-scans
      # pipelineRef:
      #   name: pip-security-scans
      taskRef:
        apiVersion: tekton.dev/v1
        kind: Pipeline
        name: pip-security-scans
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: pip-panic
spec:
  pipelineRef:
    name: pip-clone-scan-notify
  params:
    - name: test1
      value: "test1"

[jwitrick]

Using this method can avoid panic, but the occurrence of a panic is certainly not expected in the program.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: security-scans
      # pipelineRef:
      #   name: pip-security-scans
      taskRef:
        apiVersion: tekton.dev/v1
        kind: Pipeline
        name: pip-security-scans

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-security-scans
spec:
  tasks:
    - name: custom-vars-pip
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "GETTING HERE"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: custom-vars
      params:
        - name: test1
          value: $(params.test1)
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        params:
          - name: test1
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "Value of test1 is: "
              echo $(params.test1)
    - name: security-scans
      # pipelineRef:
      #   name: pip-security-scans
      taskRef:
        apiVersion: tekton.dev/v1
        kind: Pipeline
        name: pip-security-scans
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: pip-panic
spec:
  pipelineRef:
    name: pip-clone-scan-notify
  params:
    - name: test1
      value: "test1"

Nice. Thank you for a workaround at least. I really want p-i-p to work.

[jwitrick]

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
name: pip-panic
spec:
pipelineRef:
name: pip-clone-scan-notify
params:
- name: test1
value: "test1"

When I run this, I do get past the segfault, but the 2nd pipeline (pip-security-scans) never 'runs' and the pipelinerun ('pip-panic') never finishes.

Is this expected?

[l-qing]

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
name: pip-panic
spec:
pipelineRef:
name: pip-clone-scan-notify
params:

• name: test1
  value: "test1"

When I run this, I do get past the segfault, but the 2nd pipeline (pip-security-scans) never 'runs' and the pipelinerun ('pip-panic') never finishes.

Is this expected?

Yes, this requires you to install the pip controller

TL;DR

kubectl apply --filename https://storage.googleapis.com/tekton-releases-nightly/pipelines-in-pipelines/latest/release.yaml

[l-qing]

Sorry, I found out after trying that pip currently only supports v1beta1, so the yaml mentioned above has to be modified like this to be executed successfully:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: pip-security-scans
spec:
  tasks:
    - name: custom-vars-pip
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "GETTING HERE"
---
apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: pip-clone-scan-notify
spec:
  params:
  - name: test1
  tasks:
    - name: custom-vars
      params:
        - name: test1
          value: $(params.test1)
      taskSpec:
        metadata:
          annotations:
            sidecar.istio.io/inject: "false"
        params:
          - name: test1
        steps:
          - name: buildvars
            image: mikefarah/yq:4.6.1
            script: |
              #/bin/sh
              set -e
              echo "Value of test1 is: "
              echo $(params.test1)
    - name: security-scans
      # pipelineRef:
      #   name: pip-security-scans
      taskRef:
        apiVersion: tekton.dev/v1beta1
        kind: Pipeline
        name: pip-security-scans
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  name: pip-panic
spec:
  pipelineRef:
    name: pip-clone-scan-notify
  params:
    - name: test1
      value: "test1"

$ kubectl get pr -w

pip-panic                                             Unknown     Running                    7m47s
pip-panic
pip-panic
pip-panic                                             Unknown     Running                    1s
pip-panic                                             Unknown     Running                    17s
pip-panic                                             Unknown     Running                    60s
pip-panic
pip-panic
pip-panic-security-scans
pip-panic                                             Unknown     Running                    0s
pip-panic-security-scans                              Unknown     Running                    0s
pip-panic                                             Unknown     Running                    14s
pip-panic-security-scans                              True        Succeeded                  14s         0s
pip-panic                                             True        Succeeded                  14s         0s

[l-qing]

/assign l-qing