Add Task for publishing tekton pipeline images + yaml

Add a `Task` which invokes `ko` to build and publish all images and yaml
config required for installing Tekton Pipelines.

This Task will:
* Build and publish the "base image" using Kaniko
* Generate a .ko.yaml
* Invoke ko to build/publish images and generate a release.yaml
* Parse the release.yaml for built images; ensuring that the expected
  images were built (and no more)
* Tag the built images with the correct version and also tag in all
  regions (us, asia, eu)

This should be the same functionality that could previously be seen in
https://github.com/tektoncd/pipeline/blob/master/hack/release.sh
(which used
https://github.com/knative/test-infra/blob/master/scripts/release.sh).
We can remove release.sh once we have completed #530 as well.

Some functionality has been implemented in a python script, which has
its own tests. Since it is currently difficult to update the pull
request test logic to do additional things (such as run python unit
tests), I'm hoping we are okay with waiting until #532 to add
automatic running of these tests).

Use actual production bucket and registry by default (#527)

Fixes #528
Fixes #529

.gitignore

@@ -36,3 +36,6 @@
 
 # JetBrains IDE config
 .idea
+
+# Python
+*.pyc

.ko.yaml

@@ -1,5 +1,5 @@
 baseImageOverrides:
-  # TODO(jasonhall): Use build-base in the build-pipeline path, when it's build/released.
+  # TODO(christiewilson): Use our built base image
   github.com/tektoncd/pipeline/cmd/creds-init: gcr.io/knative-nightly/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/git-init: gcr.io/knative-nightly/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/bash: busybox # image should have shell in $PATH

.ko.yaml.release

@@ -1,6 +1,7 @@
 baseImageOverrides:
-  # TODO(jasonhall): Use build-base in the build-pipeline path, when it's build/released.
+  # TODO(christiewilson): Use our built base image
   github.com/tektoncd/pipeline/cmd/creds-init: gcr.io/knative-release/github.com/knative/build/build-base:latest
   github.com/tektoncd/pipeline/cmd/git-init: gcr.io/knative-release/github.com/knative/build/build-base:latest
-  github.com/tektoncd/pipeline/cmd/bash: busybox
-  github.com/tektoncd/pipeline/cmd/gsutil: google/cloud-sdk:alpine
+  github.com/tektoncd/pipeline/cmd/bash: busybox # image should have shell in $PATH
+  github.com/tektoncd/pipeline/cmd/entrypoint: busybox # image should have shell in $PATH
+  github.com/tektoncd/pipeline/cmd/gsutil: google/cloud-sdk:alpine # image should have gsutil in $PATH

docs/resources.md

@@ -309,6 +309,8 @@ spec:
       value: gcs
     - name: location
       value: gs://some-bucket
+    - name: dir
+      value: "y" # This can have any value to be considered "true"
 ```
 
 Params that can be added are the following:
@@ -361,7 +363,7 @@ service account.
        - name: location
          value: gs://some-private-bucket
        - name: dir
-         value: "directory"
+         value: "y"
      secrets:
        - fieldName: GOOGLE_APPLICATION_CREDENTIALS
          secretName: bucket-sa

hack/release.md

@@ -1,5 +1,7 @@
 # Creating a new Tekton Pipeline release
 
+**Note: we are transitioning to a Pipelines based test and release process, see [tekton/README.md](../tekton/README.md).**
+
 The `release.sh` script automates the creation of Tekton Pipeline releases,
 either nightly or versioned ones.
 

tekton/README.md

@@ -0,0 +1,71 @@
+# Tekton Repo CI/CD
+
+We dogfood our project by using Tekton Pipelines to build, test and release Tekton Pipelines!
+
+This directory contains the [`Tasks`](https://github.com/knative/build-pipeline/blob/master/docs/tasks.md)
+and [`Pipelines`](https://github.com/knative/build-pipeline/blob/master/docs/pipelines.md) that we (will)
+use.
+
+TODO(#538): In #538 or #537 we will update [Prow](https://github.com/knative/build-pipeline/blob/master/CONTRIBUTING.md#pull-request-process)
+to invoke these `Pipelines` automatically, but for now we will have to invoke them manually.
+
+## Release Pipeline
+
+The `Tasks` which make up our release `Pipeline` are:
+
+* [`publish.yaml`](publish.yaml) - This `Task` uses [`kaniko`](https://github.com/GoogleContainerTools/kaniko)
+  to build and publish base images, and uses [`ko`](https://github.com/google/go-containerregistry/tree/master/cmd/ko)
+  to build all of the container images we release and generate the `release.yaml`
+
+### Running
+
+To run these `Pipelines` and `Tasks`, you must have Tekton Pipelines installed, either via
+[an official release](https://github.com/knative/build-pipeline/blob/master/docs/install.md)
+or [from `HEAD`](https://github.com/knative/build-pipeline/blob/master/DEVELOPMENT.md#install-pipeline).
+
+TODO(#531): Add the Pipeline, for now all we have are `Tasks` which we can invoke individually
+by creating [`TaskRuns`](https://github.com/knative/build-pipeline/blob/master/docs/taskruns.md)
+and [`PipelineResources`](https://github.com/knative/build-pipeline/blob/master/docs/resources.md).
+
+TODO(#569): Normally we'd use the image `PipelineResources` to control which image registry the images are pushed to.
+However since we have so many images, all going to the same registry, we are cheating and using a parameter
+for the image registry instead.
+
+* [`publish-run.yaml`](publish-run.yaml) - This example `TaskRun` and `PipelineResources` demonstrate
+  how to invoke `publish.yaml`:
+
+  ```bash
+  kubectl apply -f tekton/publish.yaml
+  kubectl apply -f tekton/publish-run.yaml
+  ```
+
+### Authentication
+
+Users executing the publish task must be able to:
+
+* Push to the image registry (production registry is `gcr.io/tekton-releases`)
+* Write to the GCS bucket (production bucket is `gs://tekton-releases`)
+
+To be able to publish images via `kaniko` or `ko`, you must be able to push to your image registry.
+At the moment, the publish `Task` will try to use your default service account in the namespace where
+you create the `TaskRun`. If that default service account is able to push to your image registry,
+you are good to go. Otherwise, you need to use [a secret annotated with your docker registry
+credentials](https://github.com/tektoncd/pipeline/blob/master/docs/auth.md#basic-authentication-docker).
+
+TODO(#631) Ensure that we are supporting folks using credentials other than the cluster defaults; not
+sure how this will play out with publishing to our prod registry!
+
+#### Production credentials
+
+TODO(dlorenc, bobcatfish): We need to setup a group which users can be added to, as well as guidelines
+around who should be added to this group.
+
+For now, users who need access to our production registry (`gcr.io/tekton-releases`) and production
+GCS bucket (`gs://tekton-releases`) should ping @bobcatfish or @dlorenc to get added to the authorized
+users.
+
+## Supporting scripts
+
+Some supporting scripts have been written using Python 2.7:
+
+* [koparse](./koparse) - Contains logic for parsing `release.yaml` files created by `ko`

tekton/koparse/koparse.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python2.7
+
+"""
+koparse.py parses release.yaml files from `ko`
+
+The `ko` tool (https://github.com/google/go-containerregistry/tree/master/cmd/ko)
+builds images and embeds the full names of the built images in the resulting
+yaml files.
+
+This script does two things:
+
+* Parses those image names out of the release.yaml, including their digests, and
+  outputs those to stdout
+* Verifies the list of built images against an expected list, to be sure that all
+  expected images were built (and no extra images were built)
+"""
+
+import argparse
+import os
+import re
+import string
+import sys
+
+
+DIGEST_MARKER = "@sha256"
+
+
+class ImagesMismatchError(Exception):
+    def __init__(self, missing, extra):
+        self.missing = missing
+        self.extra = extra
+
+    def __str__(self):
+        errs = []
+        if self.missing:
+            errs.append("Images %s were expected but missing." % self.missing)
+        if self.extra:
+            errs.append("Images %s were present but not expected." %
+                        self.extra)
+        return string.join(errs, " ")
+
+
+class BadActualImageFormatError(Exception):
+    def __init__(self, image):
+        self.image = image
+
+    def __str__(self):
+        return "Format of image %s was unexpected, did not contain %s" % (self.image, DIGEST_MARKER)
+
+
+def parse_release(base, path):
+    """Extracts built images from the release.yaml at path
+
+    Args:
+        base: The built images will be expected to start with this string,
+            other images will be ignored
+        path: The path to the file (release.yaml) that will contain the built images
+    Returns:
+        list of the images parsed from the file
+    """
+    images = []
+    with open(path) as f:
+        for line in f:
+            match = re.search(base + ".*@sha256:[0-9a-f]*", line)
+            if match:
+                images.append(match.group(0))
+    return images
+
+
+def compare_expected_images(expected, actual):
+    """Ensures that the list of actual images includes only the expected images
+
+    Args:
+        expected: A list of all of the names of images that are expected to have
+            been built, including the path to the image without the digest
+        actual: A list of the names of the built images, including the path to the
+            image and the digest
+    """
+    for image in actual:
+        if DIGEST_MARKER not in image:
+            raise BadActualImageFormatError(image)
+
+    actual_no_digest = [string.split(image, DIGEST_MARKER)[0]
+                        for image in actual]
+
+    missing = set(expected) - set(actual_no_digest)
+    extra = set(actual_no_digest) - set(expected)
+
+    if missing or extra:
+        raise ImagesMismatchError(list(missing), list(extra))
+
+
+if __name__ == "__main__":
+    arg_parser = argparse.ArgumentParser(
+        description="Parse expected built images from a release.yaml created by `ko`")
+    arg_parser.add_argument("--path", type=str, required=True,
+                            help="Path to the release.yaml")
+    arg_parser.add_argument("--base", type=str, required=True,
+                            help="String prefix which is used to find images within the release.yaml")
+    arg_parser.add_argument("--images", type=str, required=True, nargs="+",
+                            help="List of all images expected to be built, without digests")
+    args = arg_parser.parse_args()
+
+    try:
+        images = parse_release(args.base, args.path)
+        compare_expected_images(args.images, images)
+    except (IOError, BadActualImageFormatError) as e:
+        sys.stderr.write("Error determining built images: %s\n" % e)
+        sys.exit(1)
+    except (ImagesMismatchError) as e:
+        sys.stderr.write("Expected images did not match: %s\n" % e)
+        with open(args.path) as f:
+            sys.stderr.write(f.read())
+        sys.exit(1)
+
+    print("\n".join(images))

tekton/koparse/test_koparse.py

@@ -0,0 +1,71 @@
+#!/usr/bin/env python2.7
+
+import os
+import unittest
+
+import koparse
+
+
+IMAGE_BASE = "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/"
+PATH_TO_TEST_RELEASE_YAML = os.path.join(os.path.dirname(
+    os.path.abspath(__file__)), "test_release.yaml")
+PATH_TO_WRONG_FILE = os.path.join(os.path.dirname(
+    os.path.abspath(__file__)), "koparse.py")
+BUILT_IMAGES = [
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/kubeconfigwriter@sha256:68453f5bb4b76c0eab98964754114d4f79d3a50413872520d8919a6786ea2b35",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/creds-init@sha256:67448da79e4731ab534b91df08da547bc434ab08e41d905858f2244e70290f48",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/git-init@sha256:7d5520efa2d55e1346c424797988c541327ee52ef810a840b5c6f278a9de934a",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/nop@sha256:3784d6b8f73043a29d2c1d6196801bee46fe808fbb94ba4fd21ca52dce503183",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/bash@sha256:d55917ef5c92627027e3755bfc577fbfa2fb783cccfb13a98632cb6ba6088cd6",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/gsutil@sha256:421a261436e16af4057b4a069fdae8a5aca6e37269952209ad9932a774aa0003",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/controller@sha256:bdc6f22a44944c829983c30213091b60f490b41f89577e8492f6a2936be0df41",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/webhook@sha256:cca7069a11aaf0d9d214306d456bc40b2e33e5839429bf07c123ad964d495d8a",
+]
+EXPECTED_IMAGES = [
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/kubeconfigwriter",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/creds-init",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/git-init",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/nop",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/bash",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/gsutil",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/controller",
+    "gcr.io/knative-releases/github.com/knative/build-pipeline/cmd/webhook",
+]
+
+
+class TestKoparse(unittest.TestCase):
+
+    def test_parse_release(self):
+        images = koparse.parse_release(IMAGE_BASE, PATH_TO_TEST_RELEASE_YAML)
+        self.assertListEqual(images, BUILT_IMAGES)
+
+    def test_parse_release_no_file(self):
+        with self.assertRaises(IOError):
+            koparse.parse_release(IMAGE_BASE, "whoops")
+
+    def test_parse_release_wrong_contents(self):
+        images = koparse.parse_release(IMAGE_BASE, PATH_TO_WRONG_FILE)
+        self.assertEqual(images, [])
+
+    def test_compare_expected_images(self):
+        koparse.compare_expected_images(EXPECTED_IMAGES, BUILT_IMAGES)
+
+    def test_compare_expected_images_bad_format(self):
+        with self.assertRaises(koparse.BadActualImageFormatError):
+            koparse.compare_expected_images(EXPECTED_IMAGES, EXPECTED_IMAGES)
+
+    def test_compare_expected_images_missing(self):
+        extra_expected = (EXPECTED_IMAGES[:] +
+                          ["gcr.io/knative-releases/something-else"])
+        with self.assertRaises(koparse.ImagesMismatchError):
+            koparse.compare_expected_images(extra_expected, BUILT_IMAGES)
+
+    def test_compare_expected_images_too_many(self):
+        extra_actual = (BUILT_IMAGES[:] +
+                        ["gcr.io/knative-releases/something-else@sha256:somedigest"])
+        with self.assertRaises(koparse.ImagesMismatchError):
+            koparse.compare_expected_images(EXPECTED_IMAGES, extra_actual)
+
+
+if __name__ == "__main__":
+    unittest.main()