Buildkite documentation experience

[benschwarz]

Heyo 👋

I spent some time yesterday updating a @calibreapp test suite (Rails app) to break the job into a number of steps, rather than one big, long running rake spec task.

I ended up with tabs open to a lot of different buildkite repos, the docs site, and a chat window to @toolmantim.

I've been using all of this technology for a long time, but I found the process to be confusing, and a bit of a cycle of commit→:boom:→furiously google, open 4 tabs, sometimes to tabs recently closed, scroll up and down→try again for a couple of hours. It was frustrating!

I'm hopeful that by documenting some of the things that I went through, or faced, that it'll highlight some of the work that needs to be done to improve documentation.

Background

• I'm using Buildkite Elastic stack from… a few months ago (pre 3.0 agent)
• docker-compose, a app container with linked containers for database and selenium
• pipeline.yml

A sample of the things I saw, did, or wondered

• All Buildkite examples suggested using Knapsack to break the build up. Knapsack apparently has a free plan, but I didn't know that until @toolmantim corrected me. I'd already moved on because I didn't want to evaluate or introduce another app/service/gem/whatever at the time. In contrast, I did find this blog post by Codeship that explained how to do what I wanted.
• From looking at the docker-compose plugin, I was able to figure out that it's common to use a container prebuild/cache step, it looks like you can also cache that image build to docker-hub, but you're kind of left to figure that part out for yourself.
• I wanted to use buildkite-agent artifact download, but had no idea how to most efficiently get the buildkite-agent into my container. @toolmantim helped supplied me with:

# Buildkite agent
ADD https://download.buildkite.com/agent/stable/latest/buildkite-agent-linux-amd64 /usr/local/bin/buildkite-agent
RUN chmod +x /usr/local/bin/buildkite-agent

After that, there were some undocumented environment variables that had to be "punched through" the docker-compose.yml file (This assumes you know how to do that with a docker-compose, too):

BUILDKITE_JOB_ID:
BUILDKITE_BUILD_ID:
BUILDKITE_AGENT_ACCESS_TOKEN:

• After that, the database container wasn't starting in time, this of course isn't a Buildkite issue, but it's something that could be easily described in documentation, or with an example repo. I used:

ADD https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh /usr/local/sbin/wait-for-it.sh

RUN chmod +x /usr/local/sbin/wait-for-it.sh

• From reading a docker-compose file from one of the many Buildkite repos, I'd also noticed the pattern of adding Gemfile and Gemfile.lock, bundling, then later adding the entire app. I couldn't remember exactly why people did this. It would be great to have this in BK documentation (and a blog post too). Even if you felt like this was 'well covered elsewhere', it'd likely bring some search traffic when partnered with "buildkite docker gem bundle" as a google search.

• The docker-compose plugin mentioned image-repository, but didn't explain how to authenticate the agent to do this. I raised an issue on that repo, then later found a better description of how to do this with the elastic stack repo readme.

• Overall, all Buildkite readmes I've looked at mention the: plugin: docker-compose#version stanza from pipeline.yml. Unfortunately, they all seem to fail to mention that the v3 agent is required. (Take the docker-compose plugin readme as an example of this). I tried using this stanza, then my build failed and I had to use a lesser documented environment variable instead.

• I found it difficult to find things under github.com/buildkite, then later realised there was also github.com/buildkite-plugins. This definitely made it harder to find things.

• The elastic stack repo has some great content about making docker builds faster, but if I didn't use the elastic stack, I wouldn't have stumbled across it at all. I raised an issue to track this..

In summary, here are the repos/pages that I looked at yesterday:

• https://github.com/buildkite/example-pipelines
• https://github.com/buildkite-plugins/docker-compose-buildkite-plugin
• https://buildkite.com/docs/builds/docker-containerized-builds
• https://buildkite.com/docs/agent/v3/cli-pipeline
• https://buildkite.com/docs/pipelines/defining-steps
• https://buildkite.com/docs/pipelines
• https://buildkite.com/docs/agent/v3/cli-artifact
• https://github.com/buildkite/elastic-ci-stack-for-aws
• https://buildkite.com/docs/quickstart/elastic-ci-stack-aws

Apart from that, a bunch of docker docs, searches through random blogs and lots of Googling.

Links to other open issues

• BK docs were kind of findable by typing "buildkite <thing>" into google, I saw that there was a still-open issue raised in 2016 for making docs searchable, but I wanted to +1 it from here too.
• I found two issues about undocumented environment variables, and this is a problem that I faced.
• I struggled with knowing if I should bother with the docs site (because it didn't have a lot of the answers I was looking for), or trying to navigate one of the many buildkite repos. It looks like I wasn't the first.

It'd be great if there was a guide/full recipe for a rails app (or node, python, whatever customers are using most). I really felt like I was on my own here, and it was one failure after the next… I know that's the nature of test suites, but I think some improvements to the existing docs and general standardisation would vastly improve the experience. 👍

[lox]

This is amazing, thank you @benschwarz, we'll digest.

[toolmantim]

Thanks for taking the time to document the experience! 🙏🏼

[plaindocs]

Hey folks, I think we've implemented a lot of this (search, environment variables, consolidating info in docs). <3 Closing for now.