Add nightly release pipeline 🌙 #1274

bobcatfish · 2019-09-04T21:30:36Z

Changes

This Pipeline will be triggered via prow over in the tektoncd/plumbing
repo every night. It will create releases of all images normally
released when doing official releases, plus also the image used for
building with ko, and tag them with the date and commit they were built
at, and will create the release.yaml as well.

This Pipeline is missing a few things that are in the manual release
Pipeline - due to #1124 unit tests have a race condition, due to #1205
the linting is flakey and it would be frustrating to lose a whole
nightly release, and finally due to using v0.3.1 it's not possible to
use workingDir, which is required by the golang build Task.

The Pipelines and Tasks have been updated to work with Tekton Pipelines
v0.3.1 because that's what we're using in our official cluster (since
currently Prow requires it).

Made release instructions more oriented toward someone actually
making a release vs. a random person trying to run the same pipeline
against their own infrastructure.

Removed example Runs b/c it's much simpler to invoke
via tkn, or Prow (these were falling out of date with how we were
actually using the Pipelines/Tasks as well).

Removed the gcs-uploader-image PipelineResource which is no longer
being used.

Fixes #860

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

[n/a] Includes tests (if functionality changed/added)
Includes docs (if user facing)
Commit messages follow commit message best practices

See the contribution guide for more details.

Double check this list of stuff that's easy to miss:

If you are adding a new binary/image to the cmd dir, please update
the release Task and TaskRun to build and release this image

Reviewer Notes

If API changes
are included, additive changes
must be approved by at least two OWNERS
and backwards incompatible changes
must be approved by more than 50% of the OWNERS,
and they must first be added
in a backwards compatible way.

This Pipeline will be triggered via prow over in the tektoncd/plumbing repo every night. It will create releases of all images normally released when doing official releases, plus also the image used for building with ko, and tag them with the date and commit they were built at, and will create the release.yaml as well. This Pipeline is missing a few things that are in the manual release Pipeline - due to tektoncd#1124 unit tests have a race condition, due to tektoncd#1205 the linting is flakey and it would be frustrating to lose a whole nightly release, and finally due to using v0.3.1 it's not possible to use workingDir, which is required by the golang build Task. The Pipelines and Tasks have been updated to work with Tekton Pipelines v0.3.1 because that's what we're using in our official cluster (since currently Prow requires it). Made release instructions more oriented toward someone actually making a release vs. a random person trying to run the same pipeline against their own infrastructure. Removed example Runs b/c it's much simpler to invoke via `tkn`, or Prow (these were falling out of date with how we were actually using the Pipelines/Tasks as well). Removed the `gcs-uploader-image` PipelineResource which is no longer being used. Fixes tektoncd#860

tekton-robot · 2019-09-04T21:30:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bobcatfish

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [bobcatfish]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is the configuration to run the Pipeline + Tasks being added in tektoncd/pipeline#1274 every night at midnight PDT. I've successfully run this by setting the cron to every 5 minutes. This configuration is already applied to the repo. Status can be seen at https://prow.tekton.dev/?type=periodic. Part of tektoncd/pipeline#860

vdemeester

I 😻 it 🎉

vdemeester · 2019-09-05T07:33:37Z

tekton/README.md


-1. Install / update Tekton in the kubernetes cluster you'll be running against either via:
+1. [Install Tekton v0.3.1](#install-tekton)


we should say tkn or "Tekton CLI" here right ?

vdemeester · 2019-09-05T07:46:42Z

tekton/ci-images.yaml

+        value: /secret/release.json
+
+  volumes:


This could be specified in the PipelineRun with PodTemplate (that said it's not supported yet by tkn…)

This is the configuration to run the Pipeline + Tasks being added in tektoncd/pipeline#1274 every night at midnight PDT. I've successfully run this by setting the cron to every 5 minutes. This configuration is already applied to the repo. Status can be seen at https://prow.tekton.dev/?type=periodic. Part of tektoncd/pipeline#860

dibyom

LGTM once we add the bit about installing the CLI like Vincent mentioned!

dibyom · 2019-09-05T20:17:08Z

tekton/publish.yaml

@@ -55,7 +55,7 @@ spec:
    - /kaniko/executor
    args:
    - --dockerfile=/workspace/go/src/github.com/tektoncd/pipeline/images/Dockerfile
-    - --destination=$(inputs.params.imageRegistry)/$(inputs.params.pathToProject)/$(outputs.resources.builtBaseImage.url)
+    - --destination=${inputs.params.imageRegistry}/${inputs.params.pathToProject}/${outputs.resources.builtBaseImage.url}


dibyom · 2019-09-05T20:20:30Z

tekton/resources.yaml

  - name: revision
-    value:  vX.Y.Z-invalid-tags-boouuhhh # REPLACE with your own commit
+    value:  vX.Y.Z-invalid-tags-boouuhhh # REPLACE with the commit you want to release


👍 for clear description!

afrittoli

Thank you for this! I have a few minor comments, but nothing blocking really.

afrittoli · 2019-09-11T09:27:37Z

tekton/README.md

-  gcloud container clusters get-credentials prow --zone us-central1-a --project tekton-releases
-  ```
-
+     name: tekton-pipelines-vX-Y-


Uhm, something's missing here, the rest of the pipelineresource definition is on L70-76 mixed up with the text.

afrittoli · 2019-09-11T09:29:38Z

tekton/README.md

-
-   ```shell
-   # Do not forget to change those environment variables !
+   # Change thie environment variable to the verison you would like to use.


NIT: s/verison/version

afrittoli · 2019-09-11T09:32:14Z

tekton/README.md


- [koparse](./koparse) - Contains logic for parsing `release.yaml` files created
-  by `ko`
+# Connected to the `prow` in the `tekton-releases` GCP project


NIT: prow cluster?

afrittoli · 2019-09-11T09:33:08Z

tekton/README.md

+
+## Supporting scripts and images
+
+Some supporting scripts have been written using Python 2.7:


We need to change those to python3 soon! :)

afrittoli · 2019-09-11T10:11:14Z

tekton/publish-nightly.yaml

+      mkdir -p /workspace/bucket/previous/$VERSION_TAG/
+      cp /workspace/bucket/latest/release.yaml /workspace/bucket/previous/$VERSION_TAG/release.yaml
+
+  - name: tag-images


Kaniko (the latest version at least) supports specifying multiple destinations for a single build command, meaning that we could build and tag the images in the same step. It might be that the code would be less readable though.

afrittoli · 2019-09-11T10:22:43Z

tekton/publish-nightly.yaml

@@ -0,0 +1,226 @@
+apiVersion: tekton.dev/v1alpha1


There are a few differences between publish and publish-nightly, but not so many, so it would be nice eventually to have them converge to a single Task executed with different inputs.

One difference that might be difficult to overcome is the version of the ko-ci image, which is gcr.io/tekton-releases/ko-ci in one case and gcr.io/tekton-nightly/ko-ci in the other case. I wonder if we support variable substitution in the image field as well?

vdemeester

Looks good to me as a first step, we can follow-up on this (the postsubmit job is already there it seems)
/lgtm

vdemeester · 2019-09-11T14:32:38Z

/test pull-tekton-pipeline-integration-tests

dibyom · 2019-09-12T17:19:57Z

👀

I0911 14:53:52.608] ERROR https://www.apache.org/licenses/LICENSE-2.0
I0911 14:53:52.608] Dialing to the given TCP address timed out

/test pull-tekton-pipeline-build-tests

dibyom · 2019-09-16T18:12:04Z

/test pull-tekton-pipeline-integration-tests

dibyom · 2019-09-16T18:35:49Z

/test pull-tekton-pipeline-integration-tests

tekton-robot requested review from dibyom and vdemeester September 4, 2019 21:30

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 4, 2019

googlebot added the cla: yes Trying to make the CLA bot happy with ppl from different companies work on one commit label Sep 4, 2019

tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 4, 2019

bobcatfish mentioned this pull request Sep 4, 2019

Add nightly Pipeline + try to fix some upgrade test stuff tektoncd/plumbing#69

Merged

2 tasks

vdemeester reviewed Sep 5, 2019

View reviewed changes

bobcatfish mentioned this pull request Sep 5, 2019

Update dogfood TaskRuns to use generate name #1056

Closed

dibyom reviewed Sep 5, 2019

View reviewed changes

bobcatfish mentioned this pull request Sep 6, 2019

Feature Request: Task Output Type of Param #1273

Closed

afrittoli reviewed Sep 11, 2019

View reviewed changes

tekton-robot assigned vdemeester Sep 11, 2019

vdemeester reviewed Sep 11, 2019

View reviewed changes

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Sep 11, 2019

tekton-robot merged commit 5d72079 into tektoncd:master Sep 16, 2019

dibyom mentioned this pull request Sep 16, 2019

Fix typos in release doc. #1317

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add nightly release pipeline 🌙 #1274

Add nightly release pipeline 🌙 #1274

bobcatfish commented Sep 4, 2019

tekton-robot commented Sep 4, 2019

vdemeester left a comment

vdemeester Sep 5, 2019

vdemeester Sep 5, 2019

dibyom left a comment

dibyom Sep 5, 2019

dibyom Sep 5, 2019

afrittoli left a comment

afrittoli Sep 11, 2019

afrittoli Sep 11, 2019

afrittoli Sep 11, 2019

afrittoli Sep 11, 2019

afrittoli Sep 11, 2019

afrittoli Sep 11, 2019

vdemeester left a comment

vdemeester commented Sep 11, 2019

dibyom commented Sep 12, 2019

dibyom commented Sep 16, 2019

dibyom commented Sep 16, 2019


		1. Install / update Tekton in the kubernetes cluster you'll be running against either via:
		1. [Install Tekton v0.3.1](#install-tekton)


		## Supporting scripts and images

		Some supporting scripts have been written using Python 2.7:

Add nightly release pipeline 🌙 #1274

Add nightly release pipeline 🌙 #1274

Conversation

bobcatfish commented Sep 4, 2019

Changes

Submitter Checklist

Reviewer Notes

tekton-robot commented Sep 4, 2019

vdemeester left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dibyom left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

afrittoli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vdemeester left a comment

Choose a reason for hiding this comment

vdemeester commented Sep 11, 2019

dibyom commented Sep 12, 2019

dibyom commented Sep 16, 2019

dibyom commented Sep 16, 2019