-
Notifications
You must be signed in to change notification settings - Fork 38.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add some more info to the Jenkins README. #24232
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,16 @@ | ||
# Jenkins | ||
|
||
[Jenkins](http://jenkins-ci.org/) is a pluggable continuous | ||
integration system. The Google team is running a Jenkins server on a | ||
private GCE instance for the Kubernetes project in order to run longer | ||
integration tests, continuously, on different providers. Currently, we | ||
are running tests on GCE, GKE, and AWS. | ||
integration system. The Google team is running two Jenkins servers in GCE for | ||
the Kubernetes project. The post-commit instance runs continuous builds, unit | ||
tests, integration tests, code verification tests, and end-to-end tests on | ||
multiple providers using the latest commits to the Kubernetes repo from the | ||
master and release branches. The PR Jenkins instance runs these tests on each | ||
PR by a trusted contributor, it but only runs a subset of the end-to-end tests | ||
and only on GCE. | ||
|
||
## General flow | ||
The flow of the Google Jenkins server: | ||
The flow of the post-commit Jenkins instance: | ||
* Under the `kubernetes-build` job: Every 2 minutes, Jenkins polls for a batch | ||
of new commits, after which it runs the `build.sh` script (in this directory) | ||
on the latest tip. This results in build assets getting pushed to GCS and the | ||
|
@@ -18,9 +21,7 @@ The flow of the Google Jenkins server: | |
* `kubernetes-e2e-gce`: Standard GCE e2e. | ||
* `kubernetes-e2e-gke`: GKE provider e2e, with head k8s client and GKE | ||
creating clusters at its default version. | ||
* `kubernetes-e2e-gke-ci`: GKE provider e2e, with head k8s client and GKE | ||
creating clusters at the head k8s version. | ||
* `kubernetes-e2e-aws`: AWS provider e2e. | ||
* `kubernetes-e2e-aws`: AWS provider e2e. This only runs once a day. | ||
* Each job will not run concurrently with itself, so, for instance, | ||
Jenkins executor will only ever run one `kubernetes-build` | ||
job. However, it may run the jobs in parallel, | ||
|
@@ -41,6 +42,42 @@ outside this repository, it's tricky to keep documentation for it up | |
to date quickly. However, the scripts themselves attempt to provide | ||
color for the configuration(s) that each script runs in. | ||
|
||
## GCS Log Format | ||
|
||
Our `upload-to-gcs.sh` script runs at the start and end of every job. Logs on | ||
post-commit Jenkins go under `gs://kubernetes-jenkins/logs/`. Logs on PR | ||
Jenkins go under `gs://kubernetes-jenkins-pull/pr-logs/pull/PULL_NUMBER/`. | ||
Individual run logs go into the `JOB_NAME/BUILD_NUMBER` folder. | ||
|
||
At the start of the job, it uploads `started.json` containing the version of | ||
Kubernetes under test and the timestamp. | ||
|
||
At the end, it uploads `finished.json` containing the result and timestamp, as | ||
well as the build log into `build-log.txt`. Under `artifacts/` we put our | ||
test results in `junit_XY.xml`, along with gcp resource lists and cluster logs. | ||
|
||
It also updates `latest-build.txt` at the end to point to this build number. | ||
In the end, the directory structure looks like this: | ||
|
||
``` | ||
gs://kubernetes-jenkins/logs/kubernetes-e2e-gce/ | ||
latest-build.txt | ||
12345/ | ||
build-log.txt | ||
started.json | ||
finished.json | ||
artifacts/ | ||
gcp-resources-{before, after}.txt | ||
junit_{00, 01, ...}.xml | ||
jenkins-e2e-master/{kube-apiserver.log, ...} | ||
jenkins-e2e-minion-abcd/{kubelet.log, ...} | ||
12344/ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 might be worth calling out which of these are required for munger to do its thing? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done. This is subject to change if it starts looking at started/finished.json. |
||
... | ||
``` | ||
|
||
The munger uses `latest-build.txt` and the JUnit reports to figure out whether | ||
or not the job is healthy. | ||
|
||
## Job Builder | ||
|
||
New jobs should be specified as YAML files to be processed by [Jenkins Job | ||
|
@@ -51,4 +88,11 @@ in a Docker container defined in `job-builder-image`, and triggers it using | |
[jenkins_jobs.ini](http://docs.openstack.org/infra/jenkins-job-builder/execution.html) | ||
which contains the location and credentials of the Jenkins server. | ||
|
||
E2E Job definitions are templated to avoid code duplication. To add a new job, | ||
add a new entry to the appropriate `project`. | ||
[This](https://github.com/kubernetes/kubernetes/commit/eb273e5a4bdd3905f881563ada4e6543c7eb96b5) | ||
is an example of a commit which does this. If necessary, create a new project, as in | ||
[this](https://github.com/kubernetes/kubernetes/commit/09c27cdabc300e0420a2914100bedb565c23ed73) | ||
commit. | ||
|
||
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/hack/jenkins/README.md?pixel)]() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
are you talking about separate jenkins instances or separate triggers here?
might be worth a sketch of how these are different (eg: which jobs run post-commit vs per PR)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added that above.