chromium-browser not starting #39

ardeois · 2019-09-25T20:32:19Z

Describe the bug
First I want to thank you for this image, which is really helpful for our build system.
However, since today it seems like chromium fails to start with, displaying the following error:

Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found

It looks like this happens since the last image was pushed (5 hours ago)

To Reproduce
Steps to reproduce the behavior:

Execute docker container run -it --rm --entrypoint "" zenika/alpine-chrome chromium-browser --version
See error

Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt19_Sp_make_shared_tag5_S_eqERKSt9type_info: symbol not found

I was able to reproduce the issue on multiple machines (mac and linux)

What is the expected behavior?
Chromium should be able to start and display its version

What is the actual behavior?
Chromium crashes

Possible solutions
I don't know yet but I'll try to dig in see if I find something.
If anybody knows how to fix it right away that would be greatly appreciated.

Versions

Software	Version(s)	How-to
Chromium	?	`docker run -it --rm --entrypoint "" zenika/alpine-chrome chromium-browser --version`
Image	zenika/alpine-chrome@sha256:794b317131be727d5b7489e42f28d62f400d1aa7ffade0948f0fd7942e9d27c9	`docker image inspect --format='{{index .RepoDigests 0}}' zenika/alpine-chrome`
Docker Engine	19.03.2	`docker version`

The text was updated successfully, but these errors were encountered:

binury · 2019-09-25T22:58:37Z

@ardeois The Chromium version you used (in the table ?) is 77.0.3865.75-r0.

Cloning the Dockerfile and downgrading the Chromium version eliminated this error. Didn't test many due to crunch here but found no such issue with chromium@edge=72.0.3626.121-r0.
So, issue seems to be upstream

robwilde · 2019-09-26T05:13:12Z

@teddy-error could possibly share the line you changed in the docker file to adjust the version as I am not sure of the syntax.

Thanks

jlandure · 2019-09-26T06:29:55Z

Hi @ardeois

Thank for your warm feedback 👋
I got the same error in the build done by the Docker Hub. 👌

Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found

@teddy-error Thank for your suggestion. 👍

You could do that but the best way managing docker image is to set the version of the image used using a tag like zenika/alpine-chrome:76 (available versions listed here)

For of all, we can set the version of the docker image using:

docker container run -it --rm --entrypoint "" zenika/alpine-chrome:76 chromium-browser --version

@robwilde @teddy-error 📝 I think the change is around the apk add chromium@edge (L20)
You could set a version: apk add chromium@[VERSION]
Available versions of Chromium are listed here

jlandure · 2019-09-26T10:17:31Z

For your information, the latest version installed is Chromium 77.0.3865.75-r0.
We have to investigate to check what's the problem. 🙌

mllopart · 2019-09-26T11:54:52Z

In our case we changed our docker build to

chromium@edge=72.0.3626.121-r0

That solved our problem for now in our Pipelines

jlandure · 2019-09-26T12:14:28Z

Hi @mllopart 👋
Thanks for your reply!

This is an option but you need to fork the project. 👌
I'm thinking: why are not you using the image tag zenika/alpine-chrome:72 instead of modifying the Dockerfile and building it by our own?
I am very interested! 👍

binury · 2019-09-26T19:55:47Z

@jlandure This was of course the first attempted solution (much easier than cloning after all…) but interestingly using a static version tag did not resolve the issues.

Noting for others who seem confused that the syntax to downgrade is slightly different than you wrote:
apk add chromium@edge=VERSION

ardeois · 2019-09-26T20:49:31Z

Thank you for the analysis.
I indeed got around the issue by fixing the version of the docker image. However, as we're using with-node tag, and this tag is not versioned, we had to duplicate what this Dockerfile is doing
https://github.com/Zenika/alpine-chrome/blob/0e0678f13b63556c1a6ec3a406de01ad0fc68b16/with-node/Dockerfile#L4-L9

I guess fixing this issue would have avoided that override, but it's not a big deal for now
#38

robwilde · 2019-09-26T22:30:18Z

Thanks for the info. I investigated the apk installs and did just that. I found that if I replace @edge with @v3.9 it will complete the build and install chrome 72, then everything works.
A colleague had a similar issue and suggested it actually has something to do with the Linux Kernal version 5.0.0.27 and everything works OK with 5.0.0.25.

I can confirm that as I have one build running on Elementary OS (18.04 LTS) and it works just fine with @edge version and chrome 77 but on my laptop, running POP OS (19.04 confirmed to have Kernal 5.0.0.27) it does not work until v3.9 and chrome 72
I did the test with Chrome 73 and it also fails.

Hope this helps

chadlwilson · 2019-09-28T09:00:08Z

We also ran into this with the :with-node images. Since we extend this image anyway; we could around it with a similar approach to @robwilde

FROM zenika/alpine-chrome:with-node
USER root
# Workaround for https://github.com/Zenika/alpine-chrome/issues/39
RUN apk add --no-cache chromium@edge=73.0.3683.103-r0
USER chrome

# ...

Would be great if we could get versioned variants of the with-node tags here as these images are very convenient. :-)

chadlwilson · 2019-09-29T10:34:46Z

FROM zenika/alpine-chrome:with-node@sha256:a449df361d60dddef4f5b7de8d31e6c57d7a0feef74b6ac42b777fe30b7aef85

Is also an option for those who want to use the old image in its entirety - the one that had Chromium 76 prior to its removal from Alpine's repository.

$ chromium-browser --version
Chromium 76.0.3809.132

seanpianka · 2019-09-29T17:00:52Z

@chadlwilson Why did you use USER chrome there, is there an issue using root?

gamov · 2019-09-30T00:54:02Z

FROM zenika/alpine-chrome:with-node@sha256:a449df361d60dddef4f5b7de8d31e6c57d7a0feef74b6ac42b777fe30b7aef85
Is also an option for those who want to use the old image in its entirety - the one that had Chromium 76 prior to its removal from Alpine's repository.
$ chromium-browser --version
Chromium 76.0.3809.132

Thank you for that!

chadlwilson · 2019-09-30T04:11:49Z

@chadlwilson Why did you use USER chrome there, is there an issue using root?

The base Dockerfiles run as USER chrome rather than root; so was just switching back to the same user as the base image for consistency (although running as non-root is good practice in general).

jlandure · 2019-09-30T07:22:37Z

Hi there 👋

Sorry for the delay.
Thanks @chadlwilson for your precious replies. 💚
Indeed, with-node has no version but I created the issue #38 to fix that. 👌

Has someone investigated why Chromium 77 create this error? 🤔

jlandure · 2019-09-30T07:37:50Z

For your information, I just manually published a 76-with-node version for your needs.

docker container run -it --rm --entrypoint "" zenika/alpine-chrome:76-with-node chromium-browser --version

Chromium 76.0.3809.132

chadlwilson · 2019-09-30T07:44:24Z

Thanks @jlandure - that's nicer than a direct SHA FROM instruction :-)

paretl · 2019-09-30T10:27:01Z

What about the one with the with-puppeteer tag ? This image has the exact same error with the latest version. For the moment I rebuild the image with the version 76 of Chrome.
@jlandure If you can publish a 76-with-puppeteer version it would be very great 😄
Thank you in advance!

FibreFoX · 2019-09-30T12:03:41Z

As this issue is hitting other people too, did this issue already got reported to Alpine? Do you have any issue-tracker-link or such thing?

wtayyeb · 2019-09-30T20:14:28Z

@ardeois
you need to upgrade libstdc++ to let chromium starts
apk add libstdc++@edge

FibreFoX · 2019-09-30T20:25:45Z

@wtayyeb I tried this, but did not work:

apk add --no-cache chromium@edge chromium-chromedriver@edge libstdc++@edge

Result:

Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chrome: _ZNSt19_Sp_make_shared_tag5_S_eqERKSt9type_info: symbol not found

Looks like Alpine is broken, probably due to a missing patch or so :(

EDIT:
Running chromedriver directly explodes the same way, so I assume a generic problem in the Alpine version

/ # /usr/lib/chromium/chromedriver
Error relocating /usr/lib/chromium/chromedriver: _ZNSt7__cxx1119basic_ostringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chromedriver: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found
Error relocating /usr/lib/chromium/chromedriver: _ZNSt19_Sp_make_shared_tag5_S_eqERKSt9type_info: symbol not found

EDIT 2:

I dug a bit deeper, there are similar issues on other projects (like this one https://gitlab.alpinelinux.org/alpine/aports/issues/4758) and to me it seems that there is a compability-problem, which might not be that easy to get resolved. Mixing repositories seems to be not good with Alpine-Chromium/Chromedriver anymore 😿

jlandure · 2019-10-01T09:29:07Z

Hi there!
I just tried @wtayyeb suggestion and it's working 🎉
#41

@paretl I've pushed a 76-with-puppeteer version. See here

jlandure · 2019-10-01T09:30:31Z

For your information, it's hacktoberfest. Do not hesitate to create PR to support and improve this image 🙌
More about Hacktoberfest here

paretl · 2019-10-02T09:03:29Z

Thank you @jlandure
I think it could be a great idea to have a tag with the chrome version for each one (in the future of course). It will permit to us to have a fix version of this image into our pipeline
We can talk about it if you want

* ci for centraldashboard * doc updates to ci-centraldashboard-pipeline-run.yaml * only add needed resources * snapshot * snapshot * update README.md * update README.md * update README.md * update README.md * update README.md * update numbering * update numbering * update README.md * update README.md * update README.md * ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's overlays and packaging of apps and parameters in a config file the result is a generic pipeline capability with aspects that are quite different than KF pipelines. #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes * get user's email * ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes *   **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [Kubeflow CI with tektoncd pipelines](#kubeflow-ci-with-tektoncd-pipelines) - [Use Case](#use-case) - [Pipeline Composistion](#pipeline-composistion) - [Pipeline Parameterization](#pipeline-parameterization) - [PipelineRun parameters:](#pipelinerun-parameters) - [Pipeline parameters:](#pipeline-parameters) - [Pipeline Generation](#pipeline-generation) - [Current Approach](#current-approach) - [/manifests/ci/{ci-pipeline-run, ci-pipeline}](#manifestscici-pipeline-run-ci-pipeline) - [Alternatives](#alternatives) - [Pipeline Supporting Resources](#pipeline-supporting-resources) - [Secrets](#secrets) - [Persitent Volumes](#persitent-volumes)  ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. #### PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | #### Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes *   **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [Kubeflow CI with tektoncd pipelines](#kubeflow-ci-with-tektoncd-pipelines) - [Use Case](#use-case) - [Pipeline Composition](#pipeline-composition) - [Pipeline Parameterization](#pipeline-parameterization) - [PipelineRun parameters:](#pipelinerun-parameters) - [Pipeline parameters:](#pipeline-parameters) - [Pipeline Generation](#pipeline-generation) - [Current Approach](#current-approach) - [/manifests/ci/{ci-pipeline-run, ci-pipeline}](#manifestscici-pipeline-run-ci-pipeline) - [Alternatives](#alternatives) - [Pipeline Supporting Resources](#pipeline-supporting-resources) - [Secrets](#secrets) - [Persitent Volumes](#persitent-volumes)  ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composition 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. #### PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | #### Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes * update README.md * update README.md * update README.md * address most of the comments, still need to integrate the user-gcp-sa secret * update secrets, namespace, params needed by tasks * update README to include suggestion of kustomize and vars * #!/usr/bin/env bash # # this script assumes the following # - PipelineResource of type pullRequest for kubeflow is mounted /workspace/kubeflow-<pr> # - PipelineResource of type git for kubeflow is mounted at /workspace/kubeflow # - PipelineResource of type git for manifests is mounted at /workspace/manifests # # this scripts expects the following env vars # $image_name # $pull_request_repo # $pull_request_id # # this script extracts the forked repo, branch and commit from /workspace/kubeflow-<pr>/pr.json # it extracts the github user from /workspace/kubeflow-<pr>/github/pr.json # it extracts the image digest from /workspace/${image_name}-digest which was created by the build-push task. # # the script does the following in forked manifests repo # - edits the image tag in the kustomization.yaml (its workingdir is where the component's manifest is) # - calls `make generate; make test` under manifests/tests # - if successful # - commits the changes # - creates a PR. # if [[ -z $pull_request_id || -z $pull_request_repo || -z $image_name ]]; then echo 'invalid env var pull_request_id='$pull_request_id' pull_request_repo='$pull_request_repo' image_name='$image_name exit 1 fi if [[ -f /workspace/${pull_request_repo}-${pull_request_id}/head.json ]]; then pushd /workspace/${pull_request_repo}-${pull_request_id} kubeflow_forked_repo=$(cat head.json | jq .Head.Repo? | xargs) kubeflow_forked_branch=$(cat head.json | jq .Head.Branch? | xargs) kubeflow_forked_commit=$(cat head.json | jq .Head.SHA? | xargs) new_branch_name=$(echo ${kubeflow_forked_branch}-image-update | xargs) user=$(cat github/pr.json | jq .user.login | xargs) export GITHUB_TOKEN=$(kubectl get secrets github-token-secret -ojson | jq '. | .data.token' | xargs | base64 -d) email=$(curl "https://api.github.com/users/${user}?access_token=${GITHUB_TOKEN}" -s | jq .email | xargs popd git config --global user.name $user git config --global user.email $email kubectl get secret github-secret -ojson | jq ' . | .data["ssh-publickey"]' | xargs | base64 -d > ~/.ssh/id_rsa.pub cp ~/.ssh/id_github-secret ~/.ssh/id_rsa chmod 0600 ~/.ssh/id_rsa chmod 0600 ~/.ssh/id_rsa.pub ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts ssh-keyscan -t rsa github.com > /root/.ssh/known_hosts GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git remote add upstream git@github.com:kubeflow/manifests.git GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git fetch upstream master GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git checkout -b $new_branch_name upstream/master kustomize edit set image gcr.io/kubeflow-images-public/${image_name}=gcr.io/kubeflow-images-public/${image_name}@$(cat /kubeflow/${image_name}-digest) cd /workspace/manifests/tests make generate && make test if (( $? == 0 )); then GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git commit -a -m "image updated as part of $kubeflow_forked_repo branch:$kubeflow_forked_branch PR:${pull_request_id}" GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git push origin $new_branch_name -f tmpfile=$(mktemp) echo "[auto PR] Update the $image_name image to $(cat /kubeflow/${image_name}-digest)" > $tmpfile echo "" >> $tmpfile echo "" >> $tmpfile echo "Image built from repo:$kubeflow_forked_repo branch:$kubeflow_forked_branch commit:$kubeflow_forked_commit" >> $tmpfile GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" hub pull-request -f -b 'kubeflow:master' -F $tmpfile else echo 'make generate && make test' failed fi else echo 'invalid pull-request' echo /workspace/${pull_request_repo}-${pull_request_id}/pr.json does not exist fi * update to use kustomize * update with kustomize * minor change to remove --- * update TOC in README.md * simplify approach * updates to rebuild-manifests * standardize secets name * refactored for ci use case * update README.md * fix for pull request * some small refactoring * update toc * small correction * set namespace to kubeflow-test-infra * fall back to https://github.com/Zenika/alpine-chrome/issues/39 * add TODO on role-binding

jlandure · 2019-10-02T14:38:09Z

@paretl 💡 Yes Indeed. I'm trying to do it in #38 🙌

jlandure · 2019-10-03T15:12:05Z

Hey @paretl 👋
You can now find new tags: 🎉

with-node, 77-with-node, 77-with-node-10
with-puppeteer, 77-with-puppeteer

See README or DockerHub for more info

ardeois · 2019-10-03T17:30:47Z

Thank @jlandure
I confirm using latest docker version fixed the issue.
Should I close this issue or I let you close it?

jlandure · 2019-10-03T19:22:44Z

Issue closed! Thanks everyone for your support 🎉

Do not hesitate to give information on your usage with this image using the issue template "support".
I am thinking of refactoring the readme and add a contributor page like this one

wtayyeb · 2019-10-05T19:09:22Z

@FibreFoX full installation command is like below on a pure alpine1.10 image

    apk add --no-cache chromium@edge harfbuzz@edge nss@edge freetype@edge ttf-freefont@edge libstdc++@edge

FibreFoX · 2019-10-06T09:07:51Z

@wtayyeb ;) my problems are fixed now, switched our docker-image to be based on alpine:edge, due to the problem with mixing repositores having different glibc-stuff (which ends up not being able to get mixed that easy). I got to this issue "by accident" using DuckDuckGo because I found that _ZNSt19_Sp_make_shared_tag5_S_eqERKSt9type_info: symbol not found-stuff. So your solution did not work for me, so I created a new docker-image based on the same repository instead of mixed ones.

jlandure · 2019-10-06T12:04:19Z

Hi @FibreFoX
Glad you succeed 🎉
I was wondering: why don't you update this repository to improve it and help others instead of forking or creating your own image? 🤔
It's hacktoberfest 🎉✌️

…ent))

jlandure · 2020-05-03T14:32:26Z

@all-contributors please add @ardeois for bug

allcontributors · 2020-05-03T14:32:34Z

@jlandure

I've put up a pull request to add @ardeois! 🎉

jlandure · 2020-05-03T14:33:56Z

@all-contributors please add @FibreFoX for question

allcontributors · 2020-05-03T14:34:03Z

@jlandure

I've put up a pull request to add @FibreFoX! 🎉

jlandure · 2020-05-03T14:35:39Z

@all-contributors please add @chadlwilson for question and ideas

allcontributors · 2020-05-03T14:35:47Z

@jlandure

I've put up a pull request to add @chadlwilson! 🎉

* ci for centraldashboard * doc updates to ci-centraldashboard-pipeline-run.yaml * only add needed resources * snapshot * snapshot * update README.md * update README.md * update README.md * update README.md * update README.md * update numbering * update numbering * update README.md * update README.md * update README.md * ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's overlays and packaging of apps and parameters in a config file the result is a generic pipeline capability with aspects that are quite different than KF pipelines. #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes * get user's email * ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes *   **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [Kubeflow CI with tektoncd pipelines](#kubeflow-ci-with-tektoncd-pipelines) - [Use Case](#use-case) - [Pipeline Composistion](#pipeline-composistion) - [Pipeline Parameterization](#pipeline-parameterization) - [PipelineRun parameters:](#pipelinerun-parameters) - [Pipeline parameters:](#pipeline-parameters) - [Pipeline Generation](#pipeline-generation) - [Current Approach](#current-approach) - [/manifests/ci/{ci-pipeline-run, ci-pipeline}](#manifestscici-pipeline-run-ci-pipeline) - [Alternatives](#alternatives) - [Pipeline Supporting Resources](#pipeline-supporting-resources) - [Secrets](#secrets) - [Persitent Volumes](#persitent-volumes)  ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composistion 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. #### PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | #### Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes *   **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* - [Kubeflow CI with tektoncd pipelines](#kubeflow-ci-with-tektoncd-pipelines) - [Use Case](#use-case) - [Pipeline Composition](#pipeline-composition) - [Pipeline Parameterization](#pipeline-parameterization) - [PipelineRun parameters:](#pipelinerun-parameters) - [Pipeline parameters:](#pipeline-parameters) - [Pipeline Generation](#pipeline-generation) - [Current Approach](#current-approach) - [/manifests/ci/{ci-pipeline-run, ci-pipeline}](#manifestscici-pipeline-run-ci-pipeline) - [Alternatives](#alternatives) - [Pipeline Supporting Resources](#pipeline-supporting-resources) - [Secrets](#secrets) - [Persitent Volumes](#persitent-volumes)  ## Kubeflow CI with tektoncd pipelines ### Use Case This is a proof of value showing a CI use case for the following 1. A developer makes source code changes to the centraldashboard component in his/her forked repo of kubeflow 1. The developer has a PR open 1. The developer pushes changes 1. This triggers a prow presubmit job that calls a py_func that submits [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) 1. The pipeline has 2 tasks: build-push and update-manifests. 1. Before starting the build-push task, the pipeline mounts the kubeflow repo revision refs/pull/4112/head at /workspace/kubeflow/head 1. The build-push task uses kaniko to generate a centraldashboard image and pushes it to gcr.io/kubeflow_public_images/centraldashboard 1. The build-push task makes both kubeflow and centraldashboard resources availabe to the next task 1. The update-manifests task has the kubeflow github resource, the manifests github resource, the kubeflow PR and the centraldashboard resource as inputs 1. The update-manifests does the following by running kubeflow/kubeflow/py/kubeflow/kubeflow/ci/rebuild-manifests.sh 1. Goes to the /workspace/kubeflow-4112 directory where the PR is. 1. Parses pr.json to get the forked repo, branch and sha. 1. Parses github/pr.json to get the user of the forked repo 1. sets up ~/.ssh/{id_rsa,id_rsa.pub,known_hosts} by using the secret: github-secret 1. does the following git operations on the forked repo 1. Calls `git remote add upstream git@github.com:kubeflow/manifests.git` 1. Calls `git fetch upstream master` 1. Calls `git checkout -b $new_branch_name upstream/master` 1. Calls kustomize edit set image using the image digest written to /workspace/centraldashboard.digest by build-push 1. Goes to /manifests/tests and calls `make generate; make test` 1. if the build && test is successful calls git commit; git push 1. Creates a PR. ### Pipeline Composition 1. The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is a TektonCD pipeline containing the following k8 resources: | **kind** | **name** | **type** | |:----------------: |:-----------------------------------------: |:------------: | | PipelineRun | ci-centraldashboard-pipeline-run-52fdfc07 | - | | Pipeline | ci-pipeline | - | | Task | build-push | - | | Task | update-manifests | - | | PipelineResource | centraldashboard | image | | PipelineResource | kubeflow | git | | PipelineResource | kubeflow-4091 | pullResource | | PipelineResource | manifests | git | 2. The general relationships between TektonCD resources is shown below: ``` ── PipelineRun ├── Pipeline │ └── Tasks └── PipelineResources ``` In this particular instance, the PipelineRun instance has the following composition: ``` └── ci-centraldashboard-pipeline-run-52fdfc07 ├── pipeline │ └── tasks │ ├── build-push │ └── update-manifests └── resources ├── gcr-image │ └── centraldashboard ├── gitrepo │ ├── kubeflow │ └── manifests └── pullrequest └── kubeflow#4112 ``` The Tasks within the Pipeline use Resources provided by the PipelineRun. The PipelineRun is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: labels: app.kubernetes.io/component: kubeflow app.kubernetes.io/instance: ci-centraldashboard-pipeline-run-52fdfc07 app.kubernetes.io/managed-by: kfctl app.kubernetes.io/name: ci-pipeline-run app.kubernetes.io/part-of: kubeflow app.kubernetes.io/version: v0.6 scope: kubeflow-ci name: ci-centraldashboard-pipeline-run-52fdfc07 namespace: kubeflow-ci spec: pipelineRef: name: ci-pipeline podTemplate: securityContext: runAsNonRoot: true resources: - name: kubeflow resourceRef: name: kubeflow - name: manifests resourceRef: name: manifests - name: centraldashboard resourceRef: name: centraldashboard - name: kubeflow-4091 resourceRef: name: kubeflow-4091 serviceAccount: ci-pipeline-run-service-account ``` As shown above, 3 PipelineResources are provided by the PipelineRun: - kubeflow (input) the github repo is mounted at /workspace/kubeflow - manifests (input) the github repo is mounted at /workspace/manifests - centraldashboard (output) the image is pushed to gcr - kubeflow-4112 the pullRequest is mounted at /workspace/kubeflow-4112 and holds info about the PR The Pipeline 'ci-pipeline' is shown below: ``` apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: labels: scope: kubeflow-ci name: ci-pipeline namespace: kubeflow-ci spec: params: [] resources: - name: kubeflow type: git - name: centraldashboard type: image - name: manifests type: git - name: kubeflow-4091 type: pullRequest tasks: - name: build-push params: - name: pathToDockerfile value: components/centraldashboard/Dockerfile - name: pathToContext value: components/centraldashboard - name: dockerTarget value: serve resources: inputs: - name: kubeflow resource: kubeflow outputs: - name: kubeflow resource: kubeflow - name: centraldashboard resource: centraldashboard taskRef: name: build-push - name: update-manifests params: - name: pathToManifestsDir value: common/centraldashboard/base - name: container_image value: gcr.io/kubeflow-ci/test-worker:latest resources: inputs: - from: - build-push name: kubeflow resource: kubeflow - name: manifests resource: manifests - name: kubeflow-4091 resource: kubeflow-4091 - from: - build-push name: centraldashboard resource: centraldashboard outputs: - name: manifests resource: manifests runAfter: - build-push taskRef: name: update-manifests ``` As shown above, 2 Tasks are executed by the Pipeline. These tasks consume inputs or produce outputs provided by the PipelineRun. ### Pipeline Parameterization The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) is parameterized by the following parameters that are listed under the params sections in PipelineRun and Pipeline. Changing just a few of the parameters allows a different component to be run using this pipeline. This is covered in a later section. #### PipelineRun parameters: | **name** | **value** | **description** | |:-----------------------: |:----------------------------------------------: |:-----------------------------------------: | | namespace | kubeflow-ci | namespace to run the pipeline in | | generateName | ci-centraldashboard-pipeline-run- | a suffix is added to make the name unique | | pipeline | ci-pipeline | the pipeline that the PipelineRun uses | | image_name | centraldashboard | name in the gcr | | image_url | gcr.io/kubeflow-images-public/centraldashboard | gcr url | | kubeflow_repo_revision | refs/pull/4112/head | the PR the developer is working in | | kubeflow_repo_url | git@github.com:kubeflow/kubeflow.git | the kubeflow repo | | manifests_repo_revision | master | the manifests revision | | manifests_repo_url | git@github.com:kkasravi/manifests.git | the forked manifests repo | | pull_request_repo | kubeflow | repo for the pullRequest resource | | pull_request_id | 4112 | the pullRequest id | | pvc_mount_path | kubeflow | a shared pvc for tasks to write to | | project | kubeflow-ci | the GKE project | #### Pipeline parameters: | **name** | **value** | **description** | |:---------------------: |:--------------------------------------: |:------------------------------------------: | | image_name | centraldashboard | name of the image resource | | docker_target | serve | the docker arg for multi-build dockerfiles | | path_to_context | components/centraldashboard | URI to component in kubeflow repo | | path_to_docker_file | components/centraldashboard/Dockerfile | path to Dockerfile | | path_to_manifests_dir | common/centraldashboard/base | path to kustomization.yaml to edit | | container_image | gcr.io/kubeflow-ci/test-worker:latest | image to run in container | | pull_request_repo | kubeflow | repo for pullRequest resource | | pull_request_id | 4112 | PR id for pullRequest resource | ### Pipeline Generation #### Current Approach The [ci-centraldashboard-pipeline-run.yaml](./ci-centraldashboard-pipeline-run.yaml) was generated by kfctl which used 2 manifests and a ci-centraldashboard config file. The manifests are [ci-pipeline](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline) and [ci-pipeline-run](https://github.com/kkasravi/manifests/tree/e2e-pipelineruns/ci/ci-pipeline-run). The config file is [ci-centraldashboard-pipeline-run.yaml](https://github.com/kkasravi/manifests/blob/e2e-pipelineruns/kfdef/ci-centraldashboard-pipeline-run.yaml). By leveraging kfctl's KfDef which includes overlays, and parameters per app, the result is a generic pipeline capability with aspects that are quite different than KF pipelines. ##### /manifests/ci/{ci-pipeline-run, ci-pipeline} 1. ci-pipeline The ci-pipeline manifest includes a number of overlays, each of which is a Task ``` ci-pipeline ├── base └── overlays ├── build-push-task ├── create-cluster-task ├── deploy-app-task └── update-manifests-task ``` A kfctl config file builds a pipeline by adding overlays. For ci-centraldashboard-pipeline-run.yaml the tasks are added to the ci-pipeline as overlays: ``` - kustomizeConfig: overlays: - build-push-task - update-manifests-task ``` New tasks can be added to ci-pipeline/overlays and can be composed with other tasks by adding its overlay. 1. ci-pipeline-run The ci-pipeline-run manifest also includes a number of overlay, each of which is a PipelineResource ``` ci-pipeline-run ├── base └── overlays ├── image-resource ├── kfctl-repo-resource ├── kubeflow-repo-resource ├── manifests-repo-resource ├── pull-request-resource └── testing-repo-resource ``` Overlays with repo in their name are github repos and are automounted under /workspace. Parameters for each repo include revisions that can be commit-ids, pr's, branches. The image-resource and pull-request-resource are docker images and git pull-requests respectively. These resource are also composable in any order. For ci-centraldashboard-pipeline-run.yaml the 2nd Task requires the kubeflow repo with PR, the manifests repo, the image output from build-push to get the digest, and the git pull-request. These resources are added as overlays in the ci-pipeline-run manifest: ``` - kustomizeConfig: overlays: - kubeflow-repo-resource - manifests-repo-resource - image-resource - pull-request-resource ``` #### Alternatives ### Pipeline Supporting Resources #### Secrets #### Persitent Volumes * update README.md * update README.md * update README.md * address most of the comments, still need to integrate the user-gcp-sa secret * update secrets, namespace, params needed by tasks * update README to include suggestion of kustomize and vars * #!/usr/bin/env bash # # this script assumes the following # - PipelineResource of type pullRequest for kubeflow is mounted /workspace/kubeflow-<pr> # - PipelineResource of type git for kubeflow is mounted at /workspace/kubeflow # - PipelineResource of type git for manifests is mounted at /workspace/manifests # # this scripts expects the following env vars # $image_name # $pull_request_repo # $pull_request_id # # this script extracts the forked repo, branch and commit from /workspace/kubeflow-<pr>/pr.json # it extracts the github user from /workspace/kubeflow-<pr>/github/pr.json # it extracts the image digest from /workspace/${image_name}-digest which was created by the build-push task. # # the script does the following in forked manifests repo # - edits the image tag in the kustomization.yaml (its workingdir is where the component's manifest is) # - calls `make generate; make test` under manifests/tests # - if successful # - commits the changes # - creates a PR. # if [[ -z $pull_request_id || -z $pull_request_repo || -z $image_name ]]; then echo 'invalid env var pull_request_id='$pull_request_id' pull_request_repo='$pull_request_repo' image_name='$image_name exit 1 fi if [[ -f /workspace/${pull_request_repo}-${pull_request_id}/head.json ]]; then pushd /workspace/${pull_request_repo}-${pull_request_id} kubeflow_forked_repo=$(cat head.json | jq .Head.Repo? | xargs) kubeflow_forked_branch=$(cat head.json | jq .Head.Branch? | xargs) kubeflow_forked_commit=$(cat head.json | jq .Head.SHA? | xargs) new_branch_name=$(echo ${kubeflow_forked_branch}-image-update | xargs) user=$(cat github/pr.json | jq .user.login | xargs) export GITHUB_TOKEN=$(kubectl get secrets github-token-secret -ojson | jq '. | .data.token' | xargs | base64 -d) email=$(curl "https://api.github.com/users/${user}?access_token=${GITHUB_TOKEN}" -s | jq .email | xargs popd git config --global user.name $user git config --global user.email $email kubectl get secret github-secret -ojson | jq ' . | .data["ssh-publickey"]' | xargs | base64 -d > ~/.ssh/id_rsa.pub cp ~/.ssh/id_github-secret ~/.ssh/id_rsa chmod 0600 ~/.ssh/id_rsa chmod 0600 ~/.ssh/id_rsa.pub ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts ssh-keyscan -t rsa github.com > /root/.ssh/known_hosts GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git remote add upstream git@github.com:kubeflow/manifests.git GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git fetch upstream master GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git checkout -b $new_branch_name upstream/master kustomize edit set image gcr.io/kubeflow-images-public/${image_name}=gcr.io/kubeflow-images-public/${image_name}@$(cat /kubeflow/${image_name}-digest) cd /workspace/manifests/tests make generate && make test if (( $? == 0 )); then GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git commit -a -m "image updated as part of $kubeflow_forked_repo branch:$kubeflow_forked_branch PR:${pull_request_id}" GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" git push origin $new_branch_name -f tmpfile=$(mktemp) echo "[auto PR] Update the $image_name image to $(cat /kubeflow/${image_name}-digest)" > $tmpfile echo "" >> $tmpfile echo "" >> $tmpfile echo "Image built from repo:$kubeflow_forked_repo branch:$kubeflow_forked_branch commit:$kubeflow_forked_commit" >> $tmpfile GIT_SSH_COMMAND="ssh -i ~/.ssh/id_rsa" hub pull-request -f -b 'kubeflow:master' -F $tmpfile else echo 'make generate && make test' failed fi else echo 'invalid pull-request' echo /workspace/${pull_request_repo}-${pull_request_id}/pr.json does not exist fi * update to use kustomize * update with kustomize * minor change to remove --- * update TOC in README.md * simplify approach * updates to rebuild-manifests * standardize secets name * refactored for ci use case * update README.md * fix for pull request * some small refactoring * update toc * small correction * set namespace to kubeflow-test-infra * fall back to https://github.com/Zenika/alpine-chrome/issues/39 * add TODO on role-binding

ardeois added the bug label Sep 25, 2019

kkasravi pushed a commit to kkasravi/kubeflow that referenced this issue Sep 30, 2019

fall back to jlandure/alpine-chrome#39

12fb978

jlandure mentioned this issue Oct 1, 2019

:sparkless: use only "edge" dependencies to perform update #43

Merged

jlandure closed this as completed Oct 3, 2019

GhyslainBruno added a commit to GhyslainBruno/lazyker that referenced this issue Oct 25, 2019

Changing chromium version downloaded (jlandure/alpine-chrome#39 (comm…

a48db57

…ent))

MrXinWang mentioned this issue Oct 29, 2019

Add arm64 support for centraldashboard and Fix chromium@edge version cannot start bug kubeflow/kubeflow#4422

Merged

allcontributors bot mentioned this issue May 3, 2020

docs: add ardeois as a contributor #75

Merged

allcontributors bot mentioned this issue May 3, 2020

docs: add FibreFoX as a contributor #76

Merged

allcontributors bot mentioned this issue May 3, 2020

docs: add chadlwilson as a contributor #77

Merged

chromium-browser not starting #39

chromium-browser not starting #39

Comments

ardeois commented Sep 25, 2019

binury commented Sep 25, 2019 • edited Loading

robwilde commented Sep 26, 2019

jlandure commented Sep 26, 2019

jlandure commented Sep 26, 2019

mllopart commented Sep 26, 2019

jlandure commented Sep 26, 2019 • edited Loading

binury commented Sep 26, 2019 • edited Loading

ardeois commented Sep 26, 2019

robwilde commented Sep 26, 2019

chadlwilson commented Sep 28, 2019 • edited Loading

chadlwilson commented Sep 29, 2019 • edited Loading

seanpianka commented Sep 29, 2019

gamov commented Sep 30, 2019

chadlwilson commented Sep 30, 2019

jlandure commented Sep 30, 2019

jlandure commented Sep 30, 2019

chadlwilson commented Sep 30, 2019

paretl commented Sep 30, 2019

FibreFoX commented Sep 30, 2019

wtayyeb commented Sep 30, 2019 • edited Loading

FibreFoX commented Sep 30, 2019 • edited Loading

jlandure commented Oct 1, 2019

jlandure commented Oct 1, 2019

paretl commented Oct 2, 2019

jlandure commented Oct 2, 2019

jlandure commented Oct 3, 2019 • edited Loading

ardeois commented Oct 3, 2019

jlandure commented Oct 3, 2019

wtayyeb commented Oct 5, 2019

FibreFoX commented Oct 6, 2019

jlandure commented Oct 6, 2019

jlandure commented May 3, 2020

allcontributors bot commented May 3, 2020

jlandure commented May 3, 2020

allcontributors bot commented May 3, 2020

jlandure commented May 3, 2020

allcontributors bot commented May 3, 2020

binury commented Sep 25, 2019 •

edited

Loading

jlandure commented Sep 26, 2019 •

edited

Loading

binury commented Sep 26, 2019 •

edited

Loading

chadlwilson commented Sep 28, 2019 •

edited

Loading

chadlwilson commented Sep 29, 2019 •

edited

Loading

wtayyeb commented Sep 30, 2019 •

edited

Loading

FibreFoX commented Sep 30, 2019 •

edited

Loading

jlandure commented Oct 3, 2019 •

edited

Loading