prow: introduce prow #253

nlandolfi · 2017-06-30T21:13:05Z

Introduce prow related resources and configurations.

Introduce the cluster deployment and config files for Istio
Introduce docker/prowbazel image; a modified version of the image used on Jenkins, suitable for prow jobs
Introduce test-infra-presubmit job to this repository.

Update 7/6/17: prow is running, view the simple deck frontend here. It is running as the prow cluster on istio-testing, so with access to the project one can also get the kubectl context and get the pods:

$ kubectl get pods
 NAME                     READY     STATUS    RESTARTS   AGE
deck-3409711596-8qs1f    1/1       Running   0          8m
deck-3409711596-jq191    1/1       Running   0          8m
deck-3409711596-xjdw3    1/1       Running   0          8m
hook-4091801138-bg7cr    1/1       Running   0          8m
plank-1139864133-2d0g3   1/1       Running   0          8m
sinker-79557021-1d9xm    1/1       Running   0          8m
tot-3401709015-lkwgm     1/1       Running   0          8m

This PR adds the test-infra-presubmit job file. This results of test runs from this file are here. Notice that we are piggy-backing on kubernetes instance of gubernator, a UI for viewing job results.

I have turned off public reporting for all prow jobs configured in prow/cluster/configmaps/config.yaml for the moment, because it was needlessly failing on other PRs (c.f. 254.

My intention is to create a pull request against pilot adding its job files in a similar manner. I will wait to see that those are passing (while keeping reporting off), then once that merges master we can turn on the public reporting. As we can turn on public reporting once this PR goes to master (because the jobs/test-infra-presubmit.sh file will be in master).

TODO:

Changes to bootstrap.py our k8s-test-infra fork
Look into modifying the checkout directory in bootstrap.py as per Seb's comment

googlebot · 2017-06-30T21:13:07Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for the commit author(s). If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again.

kyessenov

I hope you don't expect us to review 600k lines of code :)

nlandolfi · 2017-06-30T21:28:31Z

😂 certainly not -- advice on the right way to vendor another Git repo that we may want to modify in the future welcome...currently approaching with git subtrees ....

…

On Fri, Jun 30, 2017 at 2:16 PM Kuat ***@***.***> wrote: ***@***.**** commented on this pull request. I hope you don't expect us to review 600k lines of code :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#253 (review)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACBRUQf9Q4TaxuZMKZf0HTXo3TE75GuDks5sJWWggaJpZM4OLDo1> .

istio-testing · 2017-06-30T21:36:35Z

This is a test.

istio-testing · 2017-06-30T21:36:35Z

Jenkins job test-infra/presubmit passed

nlandolfi · 2017-06-30T21:41:38Z

for now axing k8s-test-infra subtree because I don't modify here, and we use the k8s-prow images now (rather than build our own from source).

istio-testing · 2017-06-30T21:44:44Z

This is a test.

istio-testing · 2017-06-30T21:44:45Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-06-30T22:04:04Z

This is a test.

istio-testing · 2017-06-30T22:04:04Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-06-30T22:09:03Z

This is a test.

istio-testing · 2017-06-30T22:09:04Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-06-30T22:23:44Z

This is a test.

istio-testing · 2017-06-30T22:23:45Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T22:26:03Z

This is a test.

istio-testing · 2017-07-05T22:26:04Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T22:30:37Z

This is a test.

istio-testing · 2017-07-05T22:32:24Z

This is a test.

istio-testing · 2017-07-05T22:32:30Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T22:41:56Z

This is a test.

istio-testing · 2017-07-05T22:42:41Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T22:45:12Z

This is a test.

istio-testing · 2017-07-05T22:45:51Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T22:57:33Z

This is a test.

istio-testing · 2017-07-05T22:58:10Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T23:02:54Z

This is a test.

istio-testing · 2017-07-05T23:03:00Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T23:24:14Z

This is a test.

istio-testing · 2017-07-05T23:25:01Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-05T23:35:12Z

This is a test.

nlandolfi · 2017-07-06T19:15:34Z

@istio-testing bazel test this

istio-testing · 2017-07-06T20:20:51Z

This is a test.

istio-testing · 2017-07-06T20:21:30Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-06T23:40:15Z

This is a test.

istio-testing · 2017-07-06T23:41:01Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-06T23:46:55Z

This is a test.

istio-testing · 2017-07-06T23:47:40Z

Jenkins job test-infra/presubmit passed

rshriram · 2017-07-07T02:08:41Z

docker/README.md

@@ -0,0 +1,4 @@
+This directory contains standard images maintained by Istio for test infrastructure purposes.
+
+* prowbazel - gives an environment with git, gcloud, python, bazel, and go. It is suitable for bazel-based projects.


is this deeply tied to gcloud? Can this be run on other platforms?

The reason I ask is because eventually folks would want to run tests against istio/istio on their own environments. Having a jenkins and a prow seems ripe for all sorts of issues such as test environment divergence.

The prowbazel docker image will be used in every job. We are actually uploading artifacts and logs to google cloud storage. Those would be publicly available. If folks want to upload to other cloud, they can modify this image to add new tools or create their own image. In a near future we ll probably use specific image for specific jobs.

good point @rshriram -- the prowbazel docker file in particular, not tied to GCP, just a docker file (we do host it on GCR).

to reiterate as @sebastienvas mentioned, the entrypoint of the docker file uploads logs to a gcs bucket, which is google specific. the bucket is read public.

other folks running infrastructure would likely use an image that uploads logs to their preferred storage.

ldemailly

it's exciting to see this taking shape, nice work!

quick random question below:

ldemailly · 2017-07-07T04:16:15Z

docker/prowbazel/bootstrap.py

@@ -0,0 +1,965 @@
+#!/usr/bin/env python
+
+# Copyright 2016 The Kubernetes Authors.


any way to reference rather than copy this large file ? or did you make changes to it (which change compared to the original ? where is the original?)

We have forked the kubernetest/test-infra repo here. I have a PR against our fork here. I cleaned up my changes so that they are flags which, when unspecified, do not modify current behavior, but also allow us to modify/turn off the things we need to.

ldemailly · 2017-07-07T04:18:59Z

prow/cluster/configmaps/config.yaml

@@ -150,6 +150,7 @@ presubmits:

  istio/test-infra:
  - name: test-infra-presubmit
+    skip_report: true


why ? I'm curious to see how prow reports work and I hope it doesn't have the bug that our jenkins report has of posting a comment that triggers a lot of useless github notifications ?

This diff introduces the prow/test-infra-presubmit.sh, but other PRs on these repos don't yet have the file, and the job would fail, therefore I turned off reporting.

re: notifications, Prow does comment on the PR for test failures.

sebastienvas

PTAL

sebastienvas · 2017-07-07T17:16:57Z

docker/README.md

@@ -0,0 +1,4 @@
+This directory contains standard images maintained by Istio for test infrastructure purposes.
+
+* prowbazel - gives an environment with git, gcloud, python, bazel, and go. It is suitable for bazel-based projects.


The prowbazel docker image will be used in every job. We are actually uploading artifacts and logs to google cloud storage. Those would be publicly available. If folks want to upload to other cloud, they can modify this image to add new tools or create their own image. In a near future we ll probably use specific image for specific jobs.

sebastienvas · 2017-07-07T17:19:47Z

docker/prowbazel/Dockerfile

+ENV TEST_TMPDIR /home/bootstrap/.cache/bazel
+
+# Add bootstrap, the test harness (checks out code, captures log and status)
+ADD ./docker/prowbazel/bootstrap.py ${HOME}


Based on our discussion with @ldemailly, this is where you would put an URL. https://docs.docker.com/engine/reference/builder/#add. The url might something like https://raw.githubusercontent.com/kubernetes/test-infra/master/jenkins/bootstrap.py

sebastienvas · 2017-07-07T17:21:19Z

docker/prowbazel/Makefile

+
+# Open shell to running prowbazel image for experimentation.
+run:
+	docker run -it --entrypoint bash --privileged gcr.io/istio-testing/prowbazel:0.1


replace istio-testing with $(PROJECT) and 0.1 with $(VERSION)

good catch, done.

sebastienvas · 2017-07-07T17:55:16Z

jobs/test-infra-presubmit.sh

+
+if [ "$CI" == "bootstrap" ]; then
+    # ensure correct path
+    mkdir -p $GOPATH/src/istio.io


It kind of sucks that we need to do that on every job. Could we move that feature into bootstrap once we have our own fork ?

sebastienvas · 2017-07-07T17:56:50Z

prow/README.md

+.
+├── anotherdir
+│   └── ...
+├── jobs


let s rename this to prow

sebastienvas · 2017-07-07T17:57:22Z

prow/cluster/configmaps/config.yaml

@@ -0,0 +1,102 @@
+---


let s move this to prow/config.yaml as discussed.

sebastienvas · 2017-07-07T17:57:42Z

prow/cluster/configmaps/plugins.yaml

@@ -0,0 +1,9 @@
+# Plugin repository whitelist.


let move this to prow/plugins.yaml as discussed.

kyessenov · 2017-07-07T19:57:47Z

prow/Makefile

+	kubectl apply -f cluster/deck_deployment.yaml
+
+deck-service: get-cluster-credentials
+	kubectl create -f cluster/deck_service.yaml


why not make everything apply?

good call. done

kyessenov · 2017-07-07T20:05:59Z

prow/cluster/prow_job.yaml

@@ -0,0 +1,6 @@
+apiVersion: extensions/v1beta1
+kind: ThirdPartyResource


should probably switch to CustomResourceDefinition since TPRs are deprecated. I'd expect the upstream project to do the work fairly soon.

nice catch. I created an issue upstream, I assume the plan is to make the switch.

nlandolfi · 2017-07-08T00:01:21Z

@istio-testing bazel test this

nlandolfi · 2017-07-08T00:18:29Z

@istio-testing bazel test this

nlandolfi · 2017-07-10T16:59:26Z

@istio-testing bazel test this

nlandolfi · 2017-07-10T17:08:59Z

the prow results for the presubmit are here.

@sebastienvas

googlebot · 2017-07-10T20:39:46Z

CLAs look good, thanks!

istio-testing · 2017-07-10T20:45:50Z

This is a test.

istio-testing · 2017-07-10T20:45:51Z

Jenkins job test-infra/presubmit passed

googlebot · 2017-07-10T20:55:17Z

CLAs look good, thanks!

istio-testing · 2017-07-10T20:58:45Z

This is a test.

istio-testing · 2017-07-10T20:58:46Z

Jenkins job test-infra/presubmit passed

istio-testing · 2017-07-10T20:59:33Z

@nlandolfi: I updated Prow config for you!

I updated Prow plugins config for you!.

In response to this:

Introduce prow related resources and configurations.

Introduce the cluster deployment and config files for Istio

Introduce docker/prowbazel image; a modified version of the image used on Jenkins, suitable for prow jobs

Introduce test-infra-presubmit job to this repository.

Update 7/6/17: prow is running, view the simple deck frontend here. It is running as the prow cluster on istio-testing, so with access to the project one can also get the kubectl context and get the pods:
$ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
deck-3409711596-8qs1f    1/1       Running   0          8m
deck-3409711596-jq191    1/1       Running   0          8m
deck-3409711596-xjdw3    1/1       Running   0          8m
hook-4091801138-bg7cr    1/1       Running   0          8m
plank-1139864133-2d0g3   1/1       Running   0          8m
sinker-79557021-1d9xm    1/1       Running   0          8m
tot-3401709015-lkwgm     1/1       Running   0          8m
This PR adds the test-infra-presubmit job file. This results of test runs from this file are here. Notice that we are piggy-backing on kubernetes instance of gubernator, a UI for viewing job results.

I have turned off public reporting for all prow jobs configured in prow/cluster/configmaps/config.yaml for the moment, because it was needlessly failing on other PRs (c.f. 254.

My intention is to create a pull request against pilot adding its job files in a similar manner. I will wait to see that those are passing (while keeping reporting off), then once that merges master we can turn on the public reporting. As we can turn on public reporting once this PR goes to master (because the jobs/test-infra-presubmit.sh file will be in master).

TODO:

Changes to bootstrap.py our k8s-test-infra fork

Look into modifying the checkout directory in bootstrap.py as per Seb's comment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kyessenov reviewed Jun 30, 2017

View reviewed changes

nlandolfi changed the title ~~WIP: prow: introduce prow~~ prow: introduce prow Jul 6, 2017

nlandolfi mentioned this pull request Jul 7, 2017

test-infra: add presubmit file for prow job istio/old_pilot_repo#910

Merged

rshriram reviewed Jul 7, 2017

View reviewed changes

ldemailly reviewed Jul 7, 2017

View reviewed changes

sebastienvas suggested changes Jul 7, 2017

View reviewed changes

kyessenov reviewed Jul 7, 2017

View reviewed changes

kyessenov approved these changes Jul 7, 2017

View reviewed changes

sebastienvas approved these changes Jul 10, 2017

View reviewed changes

Enable prow

c619859

add bazel {build|test}

5e2782b

nlandolfi merged commit 9a2d644 into istio:master Jul 10, 2017

		@@ -0,0 +1,4 @@
		This directory contains standard images maintained by Istio for test infrastructure purposes.

		* prowbazel - gives an environment with git, gcloud, python, bazel, and go. It is suitable for bazel-based projects.

		@@ -0,0 +1,965 @@
		#!/usr/bin/env python

		# Copyright 2016 The Kubernetes Authors.

		@@ -0,0 +1,6 @@
		apiVersion: extensions/v1beta1
		kind: ThirdPartyResource

prow: introduce prow #253

prow: introduce prow #253

Conversation

nlandolfi commented Jun 30, 2017 • edited Loading

googlebot commented Jun 30, 2017

kyessenov left a comment

Choose a reason for hiding this comment

nlandolfi commented Jun 30, 2017 via email

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

nlandolfi commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jun 30, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

istio-testing commented Jul 5, 2017

nlandolfi commented Jul 6, 2017

istio-testing commented Jul 6, 2017

istio-testing commented Jul 6, 2017

istio-testing commented Jul 6, 2017

istio-testing commented Jul 6, 2017

istio-testing commented Jul 6, 2017

istio-testing commented Jul 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldemailly left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebastienvas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nlandolfi commented Jul 8, 2017

nlandolfi commented Jul 8, 2017

nlandolfi commented Jul 10, 2017

nlandolfi commented Jul 10, 2017 • edited Loading

googlebot commented Jul 10, 2017

istio-testing commented Jul 10, 2017

istio-testing commented Jul 10, 2017

googlebot commented Jul 10, 2017

istio-testing commented Jul 10, 2017

istio-testing commented Jul 10, 2017

istio-testing commented Jul 10, 2017

@nlandolfi: I updated Prow config for you!

nlandolfi commented Jun 30, 2017 •

edited

Loading

ldemailly left a comment •

edited

Loading

nlandolfi commented Jul 10, 2017 •

edited

Loading