TektonListener and EventBinding CRD refactor proposal

[ncskier]

I'm creating this issue following the TektonListener and EventBinding code being merged into the experimental repo in the tekton-listener directory. These two CRDs add great value for eventing with Tekton, but I think they can be refactored to improve their usability.

Sorry in advance for my long-winded explanations 😄

Why I think the CRDs should be refactored

My concerns with the current TektonListener & EventBinding CRDs (documented here) are as follows:

1. The TektonListener andEventBinding store overlapping information about PipelineRuns.

First, both CRDs store the same pipelineRef field. Also, the TektonListener stores an entire PipelineRunSpec, but the EventBinding also stores the params for the PipelineRunSpec. There is a weird division where the TektonListener owns the PipelineRun information, and the EventBinding owns the PipelineResource information; however, the EventBinding creates the TektonListener, so the EventBinding must also own some of the PipelineRun information which it uses to create the TektonListener. This ownership division is confusing to me, and I think that one of the CRDs should own all of the PipelineRun information.

2. The TektonListener and EventBinding store overlapping information about eventing.

This is similar to my previous concern about ownership of information. The EventBinding has an EventRef field with an EventName and EventType. This same information is also stored in the TektonListener as its Event and EventType fields. Again, this ownership division is confusing to me, and I think that one of the CRDs should own all of the eventing information.

3. The TektonListener and EventBindings store reusable information that cannot be reused

Together, the TektonListener and EventBinding store the entire configuration of a Pipeline Execution: a PipelineRun with its necessary PipelineResources. I think that this "Pipeline Execution" information should be reusable for different types of events. However, as I wrote in my previous two points, the current configuration stores overlapping PipelineRun and eventing information in both the TektonListener and EventBinding CRDs. So, neither of these CRDs can be reused.

For example, if a user has a working TektonListener and EventBinding setup for GitHub pull_request events, and now wants to use the same Pipeline with a different repository and GitLab push events, the user will need to create new TektonListener and EventBinding CRDs. Even though the "Pipeline Execution" configuration of PipelineRun and PipelineResources will be the same for these two new CRDs, the user will not be able to reuse either of the CRDs.

4. PipelineRuns cannot use information from the event as parameters

PipelineRuns are not included in the EventBinding template section, so their parameters cannot store information from the event.

Proposal for refactoring the CRDs

My primary motivation with refactoring the CRDs is to separate the eventing information from the Pipeline Execution information, and to keep the core functionality that @iancoffey introduced intact. The following is an example of a simple Pipeline Execution triggered by a GitHub push event, using my refactored CRDs named TemplateBinding and Template (the names as well as the designs are open to discussion):

apiVersion: tekton.dev/v1alpha1
kind: TemplateBinding
metadata:
  name: simple-pipeline-push
  namespace: default
spec:
  templateRef:
    name: simple-pipeline-template
  serviceAccount: default
  event:
    class: cloudevent
    type: com.github.push
  params:
    - name: gitrevision
      value: ${event.head_commit.id}
    - name: gitrepositoryurl
      value: ${event.repository.url}
    - name: dockerusername
      value: ncskier
    - name: dockerappname
      value: helloworldapp

apiVersion: tekton.dev/v1alpha1
kind: Template
metadata:
  name: simple-pipeline-template
  namespace: default
spec:
  params:
    - name: gitrevision
      description: git revision
      default: master
    - name: gitrepositoryurl
      description: git repository url
    - name: dockerusername
      description: docker username
    - name: dockerappname
      description: docker app name
  templates:
    - kind: PipelineResource
      metadata:
        name: docker-image
      spec:
        type: image
        params:
        - name: url
          value: "docker.io/${params.dockerusername}/${params.dockerappname}"
    - kind: PipelineResource
      metadata:
        name: git-source
      spec:
        type: git
        params:
        - name: revision
          value: ${params.gitrevision}
        - name: url
          value: ${params.gitrepositoryurl}
    - kind: PipelineRun
      metadata:
        name: simple-pipeline-run
      spec:
        pipelineRef:
            name: simple-pipeline
        trigger:
          type: event
        resources:
        - name: git-source
          resourceRef:
            name: git-source
        - name: docker-image
          resourceRef:
            name: docker-image

Similar to the TektonListener CRD, each TemplateBinding CRD would create its own event listener.

Improvements

Here is how the proposed refactor addresses my previous concerns:

1. The TektonListener andEventBinding store overlapping information about PipelineRuns.

The Template owns all the information about PipelineRuns, aside from some parameters that it exposes to the TemplateBinding.

2. The TektonListener and EventBinding store overlapping information about eventing.

The TemplateBinding owns all the information about eventing.

3. The TektonListener and EventBindings store reusable information that cannot be reused

The Template stores the Pipeline Execution information that can be reused by any number of TemplateBindings. This will be especially useful when you want to create a Pipeline definition, because your Pipeline can support any type of eventing by providing a Template file along with your Pipeline and Task YAML files.

For example, if a user has a working Template and TemplateBinding setup for GitHub pull_request events, and now wants to use the same Pipeline with a different repository and GitLab push events, the user will only need to create one new TemplateBinding CRD.

4. PipelineRuns cannot use information from the event as parameters

The Template can expose a parameter used in the PipelineRun template, and the TemplateBinding can populate this parameter with information from the event.

Feedback

Please let me know what you think about this refactor proposal 🙂
@iancoffey @dlorenc

[iancoffey]

Ive read through this and there is a lot of superb information in here, thank you for it. I think I need to digest it a bit more before being able to describe my perspective on the solutions above.

My only initial takeaway is that, whatever we land on, I would like the EventBinding type to remain, as that is the most accurate description of the purpose of this experimental repo. Hopefully the EventBinding CRD can evolve a tiny bit to literally represent (and optionally create the resources for) an Event, a Binding (Listener) and take care of everything needed for them to communicate as a result of its creation.

There is def so much room for improvement in the types definitions, thanks again for this. Once Im over my cold I will drop a response of my thoughts here. Thanks!

[ncskier]

Thanks for reading it over Ian. I hope you feel better, and I look forward to hearing the rest of your thoughts on the proposal. Thanks!

[vtereso]

FWIW, I like this proposal 👍

[vincent-pli]

@iancoffey
Any plan about the refactor? or should I just try the current implement?

[iancoffey]

@vincent-pli Check out the latest PR here #25, this removes most barriers regarding trying/demoing EventBindings.

There will be a really in-depth Getting Started Guide PR'ed here soon. I think its important to see this work through before changing course too drastically, so my effort for now is 100% toward getting it all ready for usage, until next week or so. Or course, all options are on the table once things are in a good spot and Id expect some changes to the Types then.

[a-roberts]

Pretty sure we're good to close this now aren't we 😆?

Excellent work has been done by those above especially ^^ around Triggers (I'm doing an issue cleanup) 😄. For posterity and anyone new to this area, see https://github.com/tektoncd/triggers

Edit: waited 25 days and heard nothing and we're good now, closing