[mirror] Continuous build and release infrastructure
Switch branches/tags
Nothing to show
Clone or download
dmitshur cmd/gomote: use appropriate command names in usage text
The commands list and ssh had correct command names in usage text.
However, it seems the usage of create command was copied/pasted for
many other commands, and never got updated. This change fixes that.

Change-Id: I8392029c5c129bbee027c522a6e2bbfdecd07c62
Reviewed-on: https://go-review.googlesource.com/c/141687
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Latest commit c27f283 Oct 13, 2018
Permalink
Failed to load latest commit information.
app app/appengine: hide and stop testing release-branch.go1.9 Sep 6, 2018
autocertcache all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
buildenv buildenv: diagnose authentication problems in more places Aug 21, 2018
buildlet all: use Container-Optimized VMs instead of Kubernetes for buildlet c… May 11, 2018
cmd cmd/gomote: use appropriate command names in usage text Oct 14, 2018
dashboard dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders Oct 12, 2018
devapp devapp: hide Work-In-Progress reviews Oct 4, 2018
doc all: upgrade farmer.golang.org links to https Apr 3, 2017
env dashboard, env/openbsd: add OpenBSD 6.4 snapshot builders Oct 12, 2018
envutil all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
gerrit cmd/gerritbot: autoassign reviewers on changes Sep 3, 2018
internal internal/buildstats: Close bigquery and datastore clients after use Oct 1, 2018
kubernetes all: use "context" instead of golang.org/x/net/context Mar 6, 2018
livelog all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
maintner maintner: fix copy/paste errors Sep 21, 2018
pargzip all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
revdial all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
status devapp, status: start of status handler for monitoring Feb 23, 2018
tarutil all: add README.md files where missing, and tool to keep them updated Aug 2, 2017
types all: use Container-Optimized VMs instead of Kubernetes for buildlet c… May 11, 2018
vcs-test vcs-test: fix deployment and systemd socket association Aug 21, 2018
vendor/github.com/tarm/serial cmd/buildlet: output logs to serial port on windows GCE buildlets Apr 28, 2017
version version: delete generator, add README Jul 13, 2018
.dockerignore dashboard: add netbsd-8 builder Jul 14, 2017
AUTHORS add top-level info files Jan 21, 2015
CONTRIBUTING.md CONTRIBUTING.md: remove note about not accepting Pull Requests Mar 14, 2018
CONTRIBUTORS add top-level info files Jan 21, 2015
LICENSE add top-level info files Jan 21, 2015
PATENTS add top-level info files Jan 21, 2015
README.md all: use Container-Optimized VMs instead of Kubernetes for buildlet c… May 11, 2018
build.go cmd/gomote: fix gomote dialing to the coordinator after recent cert c… Apr 6, 2017
codereview.cfg build: add codereview.cfg Mar 18, 2015
go.mod go.mod: bump go-github to v17 Sep 25, 2018
go.sum go.mod: bump go-github to v17 Sep 25, 2018
update-deps.sh cmd/gitlock: new tool to lock git dependencies in place in Dockerfiles Jul 13, 2017
update-readmes.go all: add README.md files where missing, and tool to keep them updated Aug 2, 2017

README.md

Go Build Tools

This subrepository holds the source for various packages and tools that support Go's build system and the development of the Go programming language.

Report Issues / Send Patches

This repository uses Gerrit for code changes. To contribute, see https://golang.org/doc/contribute.html.

The main issue tracker for the blog is located at https://github.com/golang/go/issues. Prefix your issue with "x/build/DIR:" in the subject line.

Overview

The main components of the Go build system are:

  • The dashboard, in app/, serves https://build.golang.org/. It runs on App Engine and holds the state for which builds passed or failed, and stores the build failure logs for post-submit failures. (Trybot build failure logs are stored elsewhere). The dashboard does not execute any builds on its own.

  • The coordinator, in cmd/coordinator/, serves https://farmer.golang.org/. It runs on GKE and coordinates the whole build system. It finds work to do (both pre-submit "TryBot" work, and post-submit work) and executes builds, allocating machines to run the builds. It is the owner of all machines.

  • The Go package in buildenv/ contains constants for where the dashboard and coordinator run, for prod, staging, and local development.

  • The buildlet, in cmd/buildlet/, is the HTTP server that runs on each worker machine to execute builds on the coordinator's behalf. This runs on every possible GOOS/GOARCH value. The buildlet binaries are stored on Google Cloud Storage and fetched per-build, so we can update the buildlet binary independently of the underlying machine images. The buildlet is the most insecure server possible: it has HTTP handlers to read & write arbitrary content to disk, and to execute any file on disk. It also has an SSH tunnel handler. The buildlet must never be exposed to the Internet. The coordinator provisions buildlets in one of three ways:

    1. by creating VMs on Google Compute Engine (GCE) with custom images configured to fetch & run the buildlet on boot, listening on port 80 in a private network.

    2. by running Linux containers (on either Google Kubernetes Engine or GCE with the Container-Optimized OS image), with the container images configured to fetch & run the buildlet on start, also listening on port 80 in a private network.

    3. by taking buildlets out of a pool of connected, dedicated machines. The buildlet can run in either listen mode (as on GCE and GKE) or in reverse mode. In reverse mode, the buildlet connects out to https://farmer.golang.org/ and registers itself with the coordinator. The TCP connection is then logically reversed (using revdial and when the coordinator needs to do a build, it makes HTTP requests to the coordinator over the already-open TCP connection.

    These three pools can be viewed at the coordinator's http://farmer.golang.org/#pools

  • The env/ directory describes build environments. It contains scripts to create VM images, Dockerfiles to create Kubernetes containers, and instructions and tools for dedicated machines.

  • maintner in maintner/ is a library for slurping all of Go's GitHub and Gerrit state into memory. The daemon maintnerd in maintner/maintnerd/ runs on GKE and serves https://maintner.golang.org/. The daemon watches GitHub and Gerrit and apps to a mutation log whenever it sees new activity. The logs are stored on GCS and served to clients.

  • The godata package in maintner/godata/ provides a trivial API to let anybody write programs against Go's maintner corpus (all of our GitHub and Gerrit history), live up to the second. It takes a few seconds to load into memory and a few hundred MB of RAM after it downloads the mutation log from the network.

  • pubsubhelper in cmd/pubsubhelper/ is a dependency of maintnerd. It runs on GKE, is available at https://pubsubhelper.golang.org/, and runs an HTTP server to receive Webhook updates from GitHub on new activity and an SMTP server to receive new activity emails from Gerrit. It then is a pubsub system for maintnerd to subscribe to.

  • The gitmirror server in cmd/gitmirror/ mirrors Gerrit to GitHub, and also serves a mirror of the Gerrit code to the coordinator for builds, so we don't overwhelm Gerrit and blow our quota.

  • The Go gopherbot bot logic runs on GKE. The code is in cmd/gopherbot. It depends on maintner via the godata package.

  • The developer dashboard at https://dev.golang.org/ runs on GKE. Its code is in devapp/. It also depends on maintner via the godata package.

  • cmd/retrybuilds: a Go client program to delete build results from the dashboard

Adding a Go Builder

If you wish to run a Go builder, please email golang-dev@googlegroups.com first. There is documentation at https://golang.org/wiki/DashboardBuilders, but depending on the type of builder, we may want to run it ourselves, after you prepare an environment description (resulting in a VM image) of it. See the env directory.