Make mungegithub work on all our repos

[bgrant0607]

cc @kubernetes/contributor-experience

[foxish]

A couple of questions I had, @lavalamp @eparis, if you could answer these, it would help me out.

• How and where from are the health charts for the past week generated?
• Can we assume that the underlying mechanism by which we run tests and read log files will remain the same for other repositories? In that case, it seems like it should be sufficient to remove from code into YAML files, parameters like the Jenkins bucket-name, to make the submit-queue generic enough for other repositories.

[lavalamp]

That's in the test-infra repo, @fejta and I talked about getting that
running in the same pod as the munger.

And yes, I think we should mandate the same test result distribution
technique, just with a different bucket per repo (or maybe different
directory, but I think different bucket would be less confusing).

On Mon, Jul 11, 2016 at 10:17 AM, Anirudh Ramanathan <
notifications@github.com> wrote:

A couple of questions I had, @lavalamp https://github.com/lavalamp
@eparis https://github.com/eparis, if you could answer these, it would
help me out.

• How and where from are the health charts for the past week
  http://storage.googleapis.com/kubernetes-test-history/k8s-queue-health.svg
  generated?
• Can we assume that the underlying mechanism by which we run tests
  and read log files will remain the same for other repositories? In that
  case, it seems like it should be sufficient to remove from code into YAML
  files, parameters like the Jenkins bucket-name, to make the submit-queue
  generic enough for other repositories.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1304 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAnglkVqh8RxYSG6uEaOJqds1XbDGtUDks5qUnrGgaJpZM4JDRRf
.

[foxish]

There is a hardcoded "requiredContexts" here, and a parameter we supply from the commandline by the same name here. We use the former in some places (such as this and this) and the latter in other places. This looks to me like some inconsistency.

[lavalamp]

Yeah, those should be exposed as flags.

Also default values to our flags should be moved into the deployment.yaml
file.

On Mon, Jul 11, 2016 at 11:13 AM, Anirudh Ramanathan <
notifications@github.com> wrote:

There is a hardcoded "requiredContexts" here
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go#L72,
and a parameter we supply from the commandline here
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go#L488.
We use the former in some places (such as this
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/ok-to-test.go#L71
and this
https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/stale-green-ci.go#L85)
and the latter in other places. This looks to me like a bug.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1304 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAnglg8eVff11Qh2Ofatdf2VIwq1kpOZks5qUoe5gaJpZM4JDRRf
.

[foxish]

Okay. Does it also make sense to remove defaults from code completely and just have them in the yaml file? Some places may need some additional error handling code in that case.

[lavalamp]

Depending on the default, yes. Definitely anything with a string like
"kubernetes" in it. And yes, I wouldn't be surprised if more error handling
was needed.

On Mon, Jul 11, 2016 at 12:47 PM, Anirudh Ramanathan <
notifications@github.com> wrote:

Okay. Does it also make sense to remove defaults from code completely and
just have them in the yaml file? Some places may need some additional error
handling code in that case.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1304 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAnglpB1wfgylIjHp7_uNwWWsscOHg7bks5qUp3BgaJpZM4JDRRf
.

[foxish]

The first PR. The first one which makes the submit-queue accept GCS parameters such as bucket-name and directory from the command-line. I'm unable to add reviewers, could you please add them for me?

[bgrant0607]

@foxish Once this is working, it should assign PRs in contrib automatically. :-)

[foxish]

Which are the other repositories against which we want to start running mungegithub right away? Do any of them have jenkins-ci tests? I'm trying to define the success criteria for this issue (and have some other repository to run against).

[lavalamp]

Let's start with contrib?

It runs a travis presubmit that needs to pass. It doesn't have any jenkins
jobs (yet).

test-infra might be good to have after that (@fejta).

On Wed, Jul 13, 2016 at 7:37 AM, Anirudh Ramanathan <
notifications@github.com> wrote:

Which are the other repositories against which we want to start running
mungegithub right away? Do any of them have jenkins-ci tests? I'm trying to
define the success criteria for this issue (and have some other repository
to run against).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1304 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAngllDotqcuV1zs83tAD630i1wPJIl8ks5qVPgygaJpZM4JDRRf
.

[eparis]

contrib :)

[lavalamp]

And I think the success criteria is that you can turn up a complete set of
our tools against a new cluster in an hour, that it's clearly documented
and anyone with the keys can do it, and that it's easy to push an update to
all the deployments.

On Wed, Jul 13, 2016 at 11:04 AM, Eric Paris notifications@github.com
wrote:

contrib :)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1304 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAnglhekJKIyyCDELoCawRjSGcAJC564ks5qVSiSgaJpZM4JDRRf
.

[bgrant0607]

@foxish I'd like to run this on kubernetes.github.io.

cc @kubernetes/docs

[foxish]

This PR turns some more hardcoded options into flags. The next steps (based on my discussion with @lavalamp) are:

1. Integrate the chart generation from the test-infra repository into the submit queue deployment.yaml so that we can deploy it per submit-queue; some changes may be needed to make the generated html status pages generic.
2. Explore the use of ConfigMap and change the deployment mechanism to make deployments easier when there are multiple instances of the submit-queue.
3. Use Ingress to proxy connections from *.k8s.io to each submit-queue service.
4. Turn on the mungers incrementally for each of our repositories, starting with contrib and docs.

[bgrant0607]

Thanks, @foxish. This is super-important.

[apelisse]

@foxish @lavalamp what's left to be done for this issue to be fixed? I'd enjoy seeing a checklist

[foxish]

Things left to do at this point are:

• Deploying ConfigMaps & configuring Ingress (Deploying with configmaps #1395).
• Generate and add OWNERS files to contrib and docs repositories. (OWNERS files for each directory under contrib #1438, Adding file and directory level OWNERS for the docs repository kubernetes/website#929)
• Update documentation to ensure someone can turn up submit-queue/cherrypick against a new repository. (to add to Added more explicit instructions to locally deploy/test mungegithub #1417)
• Parameterizing queue-health containers and running them as part of the submit-queue deployment (Running queue-health containers as part of submit-queue to enable running against multiple repositories kubernetes/test-infra#306). This is being tracked as a separate issue: Running queue-health containers as part of submit-queue to enable running against multiple repositories kubernetes/test-infra#306

Later as a cleanup task, #1415 could be undertaken, but that's not essential to finishing this issue.

[foxish]

#1456 adds configmaps for docs and contrib. We should be able to spin up an instance after this.
The DNS changes appear to have propagated as well. We can start pointing things to:
http://kubernetes.submit-queue.k8s.io/

[foxish]

Closing this. The bot is currently functional on contrib and docs. There is a document describing the steps required to get the bot to function on any new repos.