Go Shell Other
bogdanap Next steps towards using a separate AMQP cluster for logs processing (#…
…478)

Having a separate logs cluster is part of our strategy for RabbitMQ scalability, so I added the minimum necessary changes in order to get something functional and be able to test.
Latest commit cd7a2fe Jul 23, 2018
Permalink
Failed to load latest commit information.
.github Move the PR template into `.github/` Aug 28, 2017
backend Merge branch 'master' into meat-docker-env-image-selection Jul 17, 2018
bin Increase thoroughness of removals in uninstaller Apr 25, 2018
cmd/travis-worker Migrate imports and vendoring for cli lib Jul 13, 2016
config Enable the use of the RabbitMQ sharding plugin for travis-logs (#464) Jun 27, 2018
context Add instance_id to StateUpdate metadata and associated tests. Add `sc… Oct 25, 2017
errors errors: return error type from NewWrappedJobAbortError Feb 13, 2017
image Ensure env-based is more like api-based image selection Jul 18, 2018
metrics Remove some unused code Feb 13, 2017
packaging Fix shellcheck errors (#430) Jan 31, 2018
ratelimit update name for sarahhodne Jun 27, 2018
script Ensure log draining handy script is executable Jul 2, 2018
ssh Changed from SSHDialer to AuthDialer Aug 16, 2017
vendor Refresh dependencies of crypto/ssh Jul 18, 2018
.codeclimate.yml codeclimate: fix syntax for ratings Jul 13, 2015
.docker-entrypoint.sh Add some smoke testing to docker image Sep 18, 2016
.dockerignore Resolve merge conflict (shfmt version) Sep 11, 2017
.editorconfig Increase tab width to 4 spaces Feb 7, 2017
.example.env Add more stuff to the README, add a couple example files Sep 8, 2017
.gitignore Address @meatballhat feedback Jul 2, 2018
.travis.yml Update all dependencies and build on go 1.10.3 Jul 18, 2018
.vagrant-provision.sh Upgrade to go 1.8.3 + update all dependency references Jun 6, 2017
CHANGELOG.md Prepare for v3.12.0 release Jul 18, 2018
CONTRIBUTING.md Minor project identity consistency adjustments Aug 10, 2015
Dockerfile Update all dependencies and build on go 1.10.3 Jul 18, 2018
LICENSE Prepare for v3.7.0 release Apr 17, 2018
Makefile Fix worker build (#458) Jun 11, 2018
README.md Remove a layer of alias lookup in image env selector Jul 17, 2018
Vagrantfile Refresh the vagrant setup May 11, 2017
amqp_canceller.go Add pointer addresses for more info in logs Aug 18, 2017
amqp_canceller_test.go get tests running again May 15, 2018
amqp_job.go Enable the use of the RabbitMQ sharding plugin for travis-logs (#464) Jun 27, 2018
amqp_job_queue.go POC of top-level vm_config key in job payload JSON Jun 29, 2018
amqp_job_test.go do not close stateUpdatePool May 15, 2018
amqp_log_writer.go Enable the use of the RabbitMQ sharding plugin for travis-logs (#464) Jun 27, 2018
amqp_log_writer_factory.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
amqp_log_writer_test.go Enable the use of the RabbitMQ sharding plugin for travis-logs (#464) Jun 27, 2018
amqp_test.go get tests running again May 15, 2018
build_script_generator.go Include source in travis-build requests Jul 20, 2017
build_script_generator_test.go replace golang.org/x/net/context with the native go 1.7 context Feb 3, 2017
canceller.go Undo some bits in http queue constructor for compat reasons Sep 20, 2017
canceller_test.go Allow multiple cancellation subscriptions for same job ID Mar 21, 2017
cli.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
cli_test.go Include "job_path" log entry if available Jun 18, 2017
doc.go Add more docs (#174) May 20, 2016
example-payload-premium.json Run the example payload JSON files through jq Jul 2, 2018
example-payload.json Run the example payload JSON files through jq Jul 2, 2018
file_job.go Ensure all contexts have JWT for state updates Oct 14, 2017
file_job_queue.go POC of top-level vm_config key in job payload JSON Jun 29, 2018
file_log_writer.go replace golang.org/x/net/context with the native go 1.7 context Feb 3, 2017
help.go Migrate imports and vendoring for cli lib Jul 13, 2016
http_job.go Build HTTP state update body more like AMQP Oct 16, 2017
http_job_queue.go POC of top-level vm_config key in job payload JSON Jun 29, 2018
http_job_queue_test.go Undo some bits in http queue constructor for compat reasons Sep 20, 2017
http_job_test.go Move HTTP job deletion responsibility to queue level Sep 25, 2017
http_log_part_sink.go Merge branch 'master' into meat-http-log-writer-nobuf Jul 22, 2017
http_log_part_sink_test.go Re-enable test for log part sink flush Jun 6, 2017
http_log_writer.go Add pointer addresses for more info in logs Aug 18, 2017
http_log_writer_test.go Remove redundant buffering layer in HTTP log handling Jul 21, 2017
job.go commit while it works Jun 29, 2018
job_queue.go Use one HTTPJobQueue per processor Aug 18, 2017
job_test.go fix typo, thanks @meatballhat Jan 12, 2017
log_writer.go name update in mother of all comments Jun 29, 2018
log_writer_factory.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
multi_source_job_queue.go Improve worker logs (#444) Mar 8, 2018
multi_source_job_queue_test.go Use one HTTPJobQueue per processor Aug 18, 2017
package.go Implement optional instance startup & upload progress reporting Jul 4, 2018
package_test.go Ensure all contexts have JWT for state updates Oct 14, 2017
processor.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
processor_pool.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
processor_test.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
sentry_logrus_hook.go Include "job_path" log entry if available Jun 18, 2017
step_check_cancellation.go Ensure all contexts have JWT for state updates Oct 14, 2017
step_generate_script.go Use processor-level context for requeue and error state updates Oct 24, 2017
step_open_log_writer.go Next steps towards using a separate AMQP cluster for logs processing (#… Jul 23, 2018
step_open_log_writer_test.go Use processor-level context for requeue and error state updates Oct 24, 2017
step_run_script.go Improve logging around requeue conditions Oct 31, 2017
step_send_received.go Use one HTTPJobQueue per processor Aug 18, 2017
step_sleep.go Use one HTTPJobQueue per processor Aug 18, 2017
step_start_instance.go Ensure progress reporting stuff is typed and factored for safety Jul 5, 2018
step_subscribe_cancellation.go Allow multiple cancellation subscriptions for same job ID Mar 21, 2017
step_transform_build_json.go Improve naming and consistence Jul 13, 2017
step_transform_build_json_test.go Improve naming and consistence Jul 13, 2017
step_update_state.go Switch from "received at" to "processed at" Jul 17, 2018
step_upload_script.go Switch from "received at" to "processed at" Jul 17, 2018
step_write_worker_info.go Implement optional instance startup & upload progress reporting Jul 4, 2018
step_write_worker_info_test.go fixup step_write_worker_info_test.go Oct 25, 2017
systemd-wrapper Fill in tfw-ized app compat bits Apr 9, 2018
systemd.service Fill in tfw-ized app compat bits Apr 9, 2018
version.go Migrate imports and vendoring for cli lib Jul 13, 2016

README.md

Worker Build Status

Worker is the component of Travis CI that will run a CI job on some form of compute instance. It's responsible for getting the bash script from travis-build, spin up the compute instance (VM, Docker container, or maybe something different), upload the bash script, run it and stream the logs back to travis-logs. It also sends state updates to travis-hub.

Installing

from binary

Find the version you wish to install on the GitHub Releases page and download either the darwin-amd64 binary for OS X or the linux-amd64 binary for Linux. No other operating systems or architectures have pre-built binaries at this time.

from package

Use the ./bin/travis-worker-install script, or take a look at the packagecloud instructions.

from source

  1. install Go v1.7+
  2. clone this down into your $GOPATH
  • mkdir -p $GOPATH/src/github.com/travis-ci
  • git clone https://github.com/travis-ci/worker $GOPATH/src/github.com/travis-ci/worker
  • cd $GOPATH/src/github.com/travis-ci/worker
  1. install gvt:
  • go get -u github.com/FiloSottile/gvt
  1. install gometalinter:
  • go get -u github.com/alecthomas/gometalinter
  • gometalinter --install
  1. install shellcheck
  2. make

Uninstalling

In the case where Travis Worker has been installed via a deb package, there is a convenience script available which may be executed like so:

curl -sSL https://raw.githubusercontent.com/travis-ci/worker/master/bin/travis-worker-uninstall |
  sudo bash -

Configuring Travis Worker

Travis Worker is configured with environment variables or command line flags via the urfave/cli library. A list of the non-dynamic flags and environment variables may be found by invoking the built-in help system:

travis-worker --help

Environment-based image selection configuration

Some backend providers supports image selection based on environment variables. The required format uses keys that are prefixed with the provider-specific prefix:

  • TRAVIS_WORKER_{UPPERCASE_PROVIDER}_IMAGE_{UPPERCASE_NAME}: contains an image name string to be used by the backend provider

The following example is for use with the Docker backend:

# matches on `dist: trusty`
export TRAVIS_WORKER_DOCKER_IMAGE_DIST_TRUSTY=travisci/ci-connie:packer-1420290255-fafafaf

# matches on `dist: bionic`
export TRAVIS_WORKER_DOCKER_IMAGE_DIST_BIONIC=registry.business.com/fancy/ubuntu:bionic

# resolves for `language: ruby`
export TRAVIS_WORKER_DOCKER_IMAGE_RUBY=registry.business.com/travisci/ci-ruby:whatever

# resolves for `group: edge` + `language: python`
export TRAVIS_WORKER_DOCKER_IMAGE_GROUP_EDGE_PYTHON=travisci/ci-garnet:packer-1530230255-fafafaf

# used when no dist, language, or group matches
export TRAVIS_WORKER_DOCKER_IMAGE_DEFAULT=travisci/ci-garnet:packer-1410230255-fafafaf

Development: Running Travis Worker locally

This section is for anyone wishing to contribute code to Worker. The code itself should have godoc-compatible docs (which can be viewed on godoc.org: https://godoc.org/github.com/travis-ci/worker), this is mainly a higher-level overview of the code.

Environment

Ensure you've defined the necessary environment variables (see .example.env).

Pull Docker images

$ docker pull travisci/ci-amethyst:packer-1504724461
$ docker tag travisci/ci-amethyst:packer-1504724461 travis:default

Configuration

For configuration, there are some things like the job-board (TRAVIS_WORKER_JOB_BOARD_URL) and travis-build (TRAVIS_WORKER_BUILD_API_URI) URLs that need to be set. These can be set to the staging values.

export TRAVIS_WORKER_JOB_BOARD_URL='https://travis-worker:API_KEY@job-board-staging.travis-ci.com'
export TRAVIS_WORKER_BUILD_API_URI='https://x:API_KEY@build-staging.travis-ci.org/script'

TRAVIS_WORKER_BUILD_API_URI can be found in the env of the job board app, e.g.: heroku config:get JOB_BOARD_BUILD_API_ORG_URL -a job-board-staging.

Images

TODO

Configuring the requested provider/backend

Each provider requires its own configuration, which must be provided via environment variables namespaced by TRAVIS_WORKER_{PROVIDER}_.

Docker

The backend should be configured to be Docker, e.g.:

export TRAVIS_WORKER_PROVIDER_NAME='docker'
export TRAVIS_WORKER_DOCKER_ENDPOINT=unix:///var/run/docker.sock        # or "tcp://localhost:4243"
export TRAVIS_WORKER_DOCKER_PRIVILEGED="false"                          # optional
export TRAVIS_WORKER_DOCKER_CERT_PATH="/etc/secret-docker-cert-stuff"   # optional

Queue configuration

File-based queue

For the queue configuration, there is a file-based queue implementation so you don't have to mess around with RabbitMQ.

You can generate a payload via the generate-job-payload.rb script on travis-scheduler:

$ heroku run -a travis-scheduler-staging script/generate-job-payload.rb <job id> > payload.json

Place the file in the $TRAVIS_WORKER_QUEUE_NAME/10-created.d/ directory, where it will be picked up by the worker.

See example-payload.json for an example payload.

AMQP-based queue

export TRAVIS_WORKER_QUEUE_TYPE='amqp'
export TRAVIS_WORKER_AMQP_URI='amqp://guest:guest@localhost'

The web interface is accessible at http://localhost:15672/

To verify your messages are being published, try:

$ rabbitmqadmin get queue=reporting.jobs.builds

Note: You will first need to install rabbitmqadmin. See http://localhost:15672/cli

See script/publish-example-payload for a script to enqueue example-payload.json.

Building and running

Run make build after making any changes. make also executes the test suite.

  1. make
  2. ${GOPATH%%:*}/bin/travis-worker

or in Docker (FIXME):

  1. docker build -t travis-worker . # or docker pull travisci/worker
  2. docker run --env-file ENV_FILE -ti travis-worker # or travisci/worker

Testing

Run make test. To run backend tests matching Docker, for example, run go test -v ./backend -test.run Docker.

Verifying and exporting configuration

To inspect the parsed configuration in a format that can be used as a base environment variable configuration, use the --echo-config flag, which will exit immediately after writing to stdout:

travis-worker --echo-config

Stopping Travis Worker

Travis Worker has two shutdown modes: Graceful and immediate. The graceful shutdown will tell the worker to not start any additional jobs, but finish the jobs it is currently running before it shuts down. The immediate shutdown will make the worker stop the jobs it's working on and requeue them, and clean up any open resources (shut down VMs, cleanly close connections, etc.)

To start a graceful shutdown, send an INT signal to the worker (for example using kill -INT). To start an immediate shutdown, send a TERM signal to the worker (for example using kill -TERM).

Go dependency management

Travis Worker is built via the standard go commands, and dependencies managed by gvt.

To work with the dependencies you need to do the following first

  • Have this repository checked out
  • Install gvt with github.com/FiloSottile/gvt

Updating existing vendored dependencies

To update and existing vendored dependency, do the following in this directory:

  • gvt update name/of/dependency e.g. gvt update github.com/pkg/sftp

Adding a new dependency

To add a new dependency, do the following:

  • gvt fetch name/of/package e.g. gvt fetch github.com/pkg/sftp

Release process

Since we want to easily keep track of worker changes, we often associate them with a version number. To find out the current version, check the changelog or run travis-worker --version. We typically use semantic versioning to determine how to increase this number.

Once you've decided what the next version number should be, update the changelog making sure you include all relevant changes that happened since the previous version was tagged. You can see these by running git diff vX.X.X...HEAD, where v.X.X.X is the name of the previous version.

Once the changelog has been updated and merged to master, the merge commit needs to be signed and manually tagged with the version number. To do this, run:

$ git tag --sign -a vX.X.X -m "Worker version vX.X.X"
$ git push origin vX.X.X

The Travis build corresponding to this push should build and upload a worker image with the new tag to Dockerhub.

The next step is to create a new Github release tag with the appropriate information from the changelog.

License and Copyright Information

See LICENSE file.

© 2018 Travis CI GmbH