Create Tekton based Pipeline which has capability to run against multiple tekton pipelines versions #373

bobcatfish · 2020-06-18T15:30:58Z

Expected Behavior

Tasks should be able to declare what versions of Tekton Pipelines they are compatible with (Allow Tasks/Pipelines to indicate versions of Tekton Pipelines they work with pipeline#2588)
The catalog test infrastructure can use this information to determine what versions of Tekton Pipelines to test against

Actual Behavior

All catalog Tasks are tested against 1 version of Tekton Pipelines, hard coded at https://github.com/tektoncd/catalog/blob/v1beta1/test/e2e-tests.sh#L19

Additional Info

Overall catalog test infrastructure proposal

Eventually we want to be able to have Tasks declare what versions they are compatible with and test against those (tektoncd#373) but in the meantime, it seems reasonable to bump the version every time we do a release (and even once we have tektoncd#373, we would want to be running tests against new versions as well - and this could have the benefit of giving us information about issues in a release early, before users notice them!)

Eventually we want to be able to have Tasks declare what versions they are compatible with and test against those (tektoncd/catalog#373) but in the meantime, it seems reasonable to bump the version used in catalog tests every time we do a release (and even once we have tektoncd/catalog#373, we would want to be running tests against new versions as well - and this could have the benefit of giving us information about issues in a release early, before users notice them!) Question: how does the person making the release ensure that this bump didn't make anything start failing? I think we should still make this part of the process before we figure this out entirely, but we might want to create a cron or something that runs all the tests nightly, and maybe have the build cop trigger the release manually to ensure there aren't any problems?

Eventually we want to be able to have Tasks declare what versions they are compatible with and test against those (#373) but in the meantime, it seems reasonable to bump the version every time we do a release (and even once we have #373, we would want to be running tests against new versions as well - and this could have the benefit of giving us information about issues in a release early, before users notice them!)

chmouel · 2020-06-23T06:51:28Z

Really cool, it would be really nice to have.

I wonder what's is the best way to do this tho. Since pipeline supports does not support per namespace install.

Perhaps something like this :

Collect all pipeline versions of all tasks in catalog
for each pipeline versions, un-install the previous and install the new pipeline version.
Run all catalog tasks that want that specifics version against it.
Redo the same for others.

Or IOW in pseudo code this may look like this :

  for version in all_unique_versions_from_all_tasks():
       if pipeline_installed() is True:
       		uninstall_pipeline()
       install_pipeline(version)
       for task in all_task_that_has_dependency_on_pipeline_version(version):
			test_task(task)

The main issue I see is that this is going to take a lot of time for test run..

Perhaps we want to start to implement the feature to only test tasks that has been changed/added first?

bobcatfish · 2020-06-23T13:33:51Z

Hey @chmouel ! There's a sketch for how this could look in the test infra design doc - i'm (slowly) working on a prototype, my biggest goal is to do this using tekton pipelines

Collect all pipeline versions of all tasks in catalog

for each pipeline versions, un-install the previous and install the new pipeline version.

For (2) I want to do that with a Pipeline that takes a version as an argument - also if we use boskos we can create multiple clusters and install whatever versions we want. If we use a completely separate GCP project for each one we're going to run out pretty quickly tho.

This is the psuedo pipeline code I've got so far:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: catalog-test
spec:
  params:
    - name: pipelines-version
      default: "v0.13.2"
      description: The verision of tekton-pipelines to install and test with
    - name: task-paths
      description: A comma separated lists of the paths to the catalog tasks to test
  workspaces:
    - name: kubeconfig
    - name: project-creds
    - name: cluster-creds
  tasks:
    - name: get-project
      taskRef:
        name: boskos
      workspaces:
        - name: project-creds
          workspace: project-creds
    - name: make-cluster
      params:
        - name: project-name
          value: $(tasks.get-boskos-project.project-name)
      taskRef:
        name: tbd
      # Post https://github.com/tektoncd/pipeline/issues/1878 indicate workspace dependency instead
      runAfter: [get-project]
      workspaces:
        - name: kubeconfig
          workspace: kubeconfig
        - name: project-creds
          workspace: project-creds
        - name: cluster-creds
          workspace: cluster-creds
    - name: install-pipelines
      taskRef:
        name: kubectl
      # Post https://github.com/tektoncd/pipeline/issues/1878 indicate workspace dependency instead
      runAfter: [make-cluster]
      workspaces:
        - name: kubeconfig
          workspace: kubeconfig
        - name: creds
          workspace: cluster-creds
    - name: apply-tasks
      taskRef:
        name: kubectl
      runAfter: [install-pipelines]
      workspaces:
        - name: kubeconfig
          workspace: kubeconfig
        - name: creds
          workspace: cluster-creds
    - name: release-project
      taskRef:
        name: boskos
      workspaces:
        - name: project-creds
          workspace: project-creds

My next step is to start creating some of the individual tasks we'd need, e.g. the boskos task.

Eventually we want to be able to have Tasks declare what versions they are compatible with and test against those (tektoncd/catalog#373) but in the meantime, it seems reasonable to bump the version used in catalog tests every time we do a release (and even once we have tektoncd/catalog#373, we would want to be running tests against new versions as well - and this could have the benefit of giving us information about issues in a release early, before users notice them!) Question: how does the person making the release ensure that this bump didn't make anything start failing? I think we should still make this part of the process before we figure this out entirely, but we might want to create a cron or something that runs all the tests nightly, and maybe have the build cop trigger the release manually to ensure there aren't any problems?

Boskos is a tool that allows one to create a pool of cloud projects (definitely GCP, I think it supports other providers as well), and manages acquiring, releasing, and cleaning them between leases. We use it for Tekton test infrastructure for our end to end tests and we'd like to use it for our catalog Tasks as well. This commit adds boskos acquire and release Tasks. The acquire Task also creates a pod in the running cluster to perform heartbeating so that boskos knows that the resource is still in use. The intention of the release Task is that it would be run in a Pipeline's `finally` clause, however today that would be difficult because finally Tasks can't yet use the results of other Tasks, but this functionality is on the way: tektoncd/pipeline#2557 This is part of the work in tektoncd#373 to create a Pipeline for the catalog.

As part of tektoncd#373 I'm continuing to create Tasks that we can use to run tests for items in the catalog. This Task creates a cluster within a GKE project and is based on the way we create clusters for the existing end to end tests we have in Tekton, which are themselves based on Knative, when are THEMselves based on k8s. These tests all use a tool called kubetest, which is responsible for invoking boskos (see tektoncd#408) and creating a GKE cluster (https://github.com/kubernetes/test-infra/tree/master/kubetest). This Task attempts to create a cluster in the same way, which based on the output in the end to end logs seems to consist of creating the cluster and setting up a firewall for it. I'm not sure if this belongs in the catalog itself since it is specific to our own end to end tests and isn't as generic as other catalog tests, so if we want to move this into the test directory that seems fine, but I also think it could be an okay addition to the catalog on its own (esp if other k8s based projects that test against GKE want to use it.)

Boskos is a tool that allows one to create a pool of cloud projects (definitely GCP, I think it supports other providers as well), and manages acquiring, releasing, and cleaning them between leases. We use it for Tekton test infrastructure for our end to end tests and we'd like to use it for our catalog Tasks as well. This commit adds boskos acquire and release Tasks. The acquire Task also creates a pod in the running cluster to perform heartbeating so that boskos knows that the resource is still in use. The intention of the release Task is that it would be run in a Pipeline's `finally` clause, however today that would be difficult because finally Tasks can't yet use the results of other Tasks, but this functionality is on the way: tektoncd/pipeline#2557 This is part of the work in tektoncd#373 to create a Pipeline for the catalog.

As part of tektoncd#373 I'm continuing to create Tasks that we can use to run tests for items in the catalog. This Task creates a cluster within a GKE project and is based on the way we create clusters for the existing end to end tests we have in Tekton, which are themselves based on Knative, when are THEMselves based on k8s. These tests all use a tool called kubetest, which is responsible for invoking boskos (see tektoncd#408) and creating a GKE cluster (https://github.com/kubernetes/test-infra/tree/master/kubetest). This Task attempts to create a cluster in the same way, which based on the output in the end to end logs seems to consist of creating the cluster and setting up a firewall for it. I'm not sure if this belongs in the catalog itself since it is specific to our own end to end tests and isn't as generic as other catalog tests, so if we want to move this into the test directory that seems fine, but I also think it could be an okay addition to the catalog on its own (esp if other k8s based projects that test against GKE want to use it.)

As part of #373 I'm continuing to create Tasks that we can use to run tests for items in the catalog. This Task creates a cluster within a GKE project and is based on the way we create clusters for the existing end to end tests we have in Tekton, which are themselves based on Knative, when are THEMselves based on k8s. These tests all use a tool called kubetest, which is responsible for invoking boskos (see #408) and creating a GKE cluster (https://github.com/kubernetes/test-infra/tree/master/kubetest). This Task attempts to create a cluster in the same way, which based on the output in the end to end logs seems to consist of creating the cluster and setting up a firewall for it. I'm not sure if this belongs in the catalog itself since it is specific to our own end to end tests and isn't as generic as other catalog tests, so if we want to move this into the test directory that seems fine, but I also think it could be an okay addition to the catalog on its own (esp if other k8s based projects that test against GKE want to use it.)

Boskos is a tool that allows one to create a pool of cloud projects (definitely GCP, I think it supports other providers as well), and manages acquiring, releasing, and cleaning them between leases. We use it for Tekton test infrastructure for our end to end tests and we'd like to use it for our catalog Tasks as well. This commit adds boskos acquire and release Tasks. The acquire Task also creates a pod in the running cluster to perform heartbeating so that boskos knows that the resource is still in use. The intention of the release Task is that it would be run in a Pipeline's `finally` clause, however today that would be difficult because finally Tasks can't yet use the results of other Tasks, but this functionality is on the way: tektoncd/pipeline#2557 This is part of the work in #373 to create a Pipeline for the catalog.

@sbwsg

As part of tektoncd#373 I'm continuing to create Tasks that we can use to run tests for items in the catalog. The goal of the 2 Tasks in this commit is to install Tekton Pipelines into a cluster that was created using the Task added in tektoncd#436. There are a couple of options different from what I did in this commit which I'd like people's opinions on: 1. I went to a lot of trouble to make it so that Pipelines can be installed with just a kubectl command, specifically creating a user with cluster admin priviledges. If instead I used a container with gcloud, I could use the same approach as other GKE specific Tasks, but I wanted to make the install Task not tied to GKE. I also had to give the prow-account more priviledges in all of our boskos projects for this to work :grimace:. So my question is: is this worth it, or am I better off just creating a GKE specific Task for installing Tekton Pipelines that uses gcloud to auth? 2. If we DO want a super generic Task, how useful is this Task that just applies a Tekton yaml? Would we be better off with a generic kubectl task that I could feed the exact arguments for installing Tekton Pipelines? The other confusion I ran into here (which I am going to follow up with @sbwsg about!) is around service accounts vs. secrets. The GKE Tasks I've made so far rely on secrets provided via workspaces, but I feel like it's not clear if it makes sense to go that route, or to rely on service accounts. In the meantime this approach is at least consistent with tektoncd#436.

@sbwsg

As part of tektoncd#373 I'm continuing to create Tasks that we can use to run tests for items in the catalog. The goal of the 2 Tasks in this commit is to install Tekton Pipelines into a cluster that was created using the Task added in tektoncd#436. There are a couple of options different from what I did in this commit which I'd like people's opinions on: 1. I went to a lot of trouble to make it so that Pipelines can be installed with just a kubectl command, specifically creating a user with cluster admin priviledges. If instead I used a container with gcloud, I could use the same approach as other GKE specific Tasks, but I wanted to make the install Task not tied to GKE. I also had to give the prow-account more priviledges in all of our boskos projects for this to work :grimace:. So my question is: is this worth it, or am I better off just creating a GKE specific Task for installing Tekton Pipelines that uses gcloud to auth? 2. If we DO want a super generic Task, how useful is this Task that just applies a Tekton yaml? Would we be better off with a generic kubectl task that I could feed the exact arguments for installing Tekton Pipelines? The other confusion I ran into here (which I am going to follow up with @sbwsg about!) is around service accounts vs. secrets. The GKE Tasks I've made so far rely on secrets provided via workspaces, but I feel like it's not clear if it makes sense to go that route, or to rely on service accounts. In the meantime this approach is at least consistent with tektoncd#436.

tekton-robot · 2020-08-14T23:46:18Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

tekton-robot · 2020-08-14T23:46:18Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

tekton-robot · 2020-08-14T23:46:20Z

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vdemeester · 2020-08-17T08:41:59Z

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

tekton-robot · 2020-08-17T08:42:01Z

@vdemeester: Reopened this issue.

In response to this:

/remove-lifecycle rotten
/remove-lifecycle stale
/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tekton-robot · 2020-11-15T09:19:14Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

bobcatfish · 2020-11-16T21:10:12Z

I'm going to close this since I've put the WIP TEP around verified catalog entries to rest for now: tektoncd/community#170

bobcatfish self-assigned this Jun 18, 2020

bobcatfish added the kind/feature Categorizes issue or PR as related to a new feature. label Jun 18, 2020

bobcatfish mentioned this issue Jun 18, 2020

Update automation to test against 0.13.2 #374

Merged

1 task

bobcatfish mentioned this issue Jun 18, 2020

Add bumping catalog dependency to process of creating a release tektoncd/pipeline#2833

Merged

3 tasks

chmouel mentioned this issue Jun 23, 2020

Test tasks that only has been modified or added #385

Merged

3 tasks

vdemeester added the area/test-infra Issues or PRs related to the testing infrastructure label Jun 24, 2020

bobcatfish mentioned this issue Jul 9, 2020

Default hub items to "community" until we add more tests tektoncd/hub#34

Closed

bobcatfish mentioned this issue Jul 10, 2020

Add Tasks to acquire and release boskos resources 🐑 #408

Merged

2 tasks

bobcatfish mentioned this issue Jul 22, 2020

Add a task for creating a GKE cluster for e2e tests 🧪 #436

Merged

9 tasks

bobcatfish mentioned this issue Aug 5, 2020

Add Tasks to install Tekton Pipelines itself 🤯 #473

Closed

9 tasks

bobcatfish mentioned this issue Aug 13, 2020

Create a Task that can create a Tekton Pipelines cluster to test against tektoncd/plumbing#171

Closed

tekton-robot added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 14, 2020

tekton-robot closed this as completed Aug 14, 2020

tekton-robot reopened this Aug 17, 2020

tekton-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Aug 17, 2020

tekton-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Nov 15, 2020

bobcatfish closed this as completed Nov 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create Tekton based Pipeline which has capability to run against multiple tekton pipelines versions #373

Create Tekton based Pipeline which has capability to run against multiple tekton pipelines versions #373

bobcatfish commented Jun 18, 2020

chmouel commented Jun 23, 2020 •

edited

Loading

bobcatfish commented Jun 23, 2020

tekton-robot commented Aug 14, 2020

tekton-robot commented Aug 14, 2020

tekton-robot commented Aug 14, 2020

vdemeester commented Aug 17, 2020

tekton-robot commented Aug 17, 2020

tekton-robot commented Nov 15, 2020

bobcatfish commented Nov 16, 2020

Create Tekton based Pipeline which has capability to run against multiple tekton pipelines versions #373

Create Tekton based Pipeline which has capability to run against multiple tekton pipelines versions #373

Comments

bobcatfish commented Jun 18, 2020

Expected Behavior

Actual Behavior

Additional Info

chmouel commented Jun 23, 2020 • edited Loading

bobcatfish commented Jun 23, 2020

tekton-robot commented Aug 14, 2020

tekton-robot commented Aug 14, 2020

tekton-robot commented Aug 14, 2020

vdemeester commented Aug 17, 2020

tekton-robot commented Aug 17, 2020

tekton-robot commented Nov 15, 2020

bobcatfish commented Nov 16, 2020

chmouel commented Jun 23, 2020 •

edited

Loading