Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buildkite documentation experience #263

benschwarz opened this issue May 25, 2018 · 2 comments


3 participants
Copy link

commented May 25, 2018

Heyo 馃憢

I spent some time yesterday updating a @calibreapp test suite (Rails app) to break the job into a number of steps, rather than one big, long running rake spec task.

I ended up with tabs open to a lot of different buildkite repos, the docs site, and a chat window to @toolmantim.

I've been using all of this technology for a long time, but I found the process to be confusing, and a bit of a cycle of commit鈫:boom:鈫抐uriously google, open 4 tabs, sometimes to tabs recently closed, scroll up and down鈫抰ry again for a couple of hours. It was frustrating!

I'm hopeful that by documenting some of the things that I went through, or faced, that it'll highlight some of the work that needs to be done to improve documentation.


  • I'm using Buildkite Elastic stack from鈥 a few months ago (pre 3.0 agent)
  • docker-compose, a app container with linked containers for database and selenium
  • pipeline.yml

A sample of the things I saw, did, or wondered

  • All Buildkite examples suggested using Knapsack to break the build up. Knapsack apparently has a free plan, but I didn't know that until @toolmantim corrected me. I'd already moved on because I didn't want to evaluate or introduce another app/service/gem/whatever at the time. In contrast, I did find this blog post by Codeship that explained how to do what I wanted.
  • From looking at the docker-compose plugin, I was able to figure out that it's common to use a container prebuild/cache step, it looks like you can also cache that image build to docker-hub, but you're kind of left to figure that part out for yourself.
  • I wanted to use buildkite-agent artifact download, but had no idea how to most efficiently get the buildkite-agent into my container. @toolmantim helped supplied me with:
# Buildkite agent
ADD /usr/local/bin/buildkite-agent
RUN chmod +x /usr/local/bin/buildkite-agent

After that, there were some undocumented environment variables that had to be "punched through" the docker-compose.yml file (This assumes you know how to do that with a docker-compose, too):

  • After that, the database container wasn't starting in time, this of course isn't a Buildkite issue, but it's something that could be easily described in documentation, or with an example repo. I used:
ADD /usr/local/sbin/

RUN chmod +x /usr/local/sbin/
  • From reading a docker-compose file from one of the many Buildkite repos, I'd also noticed the pattern of adding Gemfile and Gemfile.lock, bundling, then later adding the entire app. I couldn't remember exactly why people did this. It would be great to have this in BK documentation (and a blog post too). Even if you felt like this was 'well covered elsewhere', it'd likely bring some search traffic when partnered with "buildkite docker gem bundle" as a google search.

  • The docker-compose plugin mentioned image-repository, but didn't explain how to authenticate the agent to do this. I raised an issue on that repo, then later found a better description of how to do this with the elastic stack repo readme.

  • Overall, all Buildkite readmes I've looked at mention the: plugin: docker-compose#version stanza from pipeline.yml. Unfortunately, they all seem to fail to mention that the v3 agent is required. (Take the docker-compose plugin readme as an example of this). I tried using this stanza, then my build failed and I had to use a lesser documented environment variable instead.

  • I found it difficult to find things under, then later realised there was also This definitely made it harder to find things.

  • The elastic stack repo has some great content about making docker builds faster, but if I didn't use the elastic stack, I wouldn't have stumbled across it at all. I raised an issue to track this..

In summary, here are the repos/pages that I looked at yesterday:

Apart from that, a bunch of docker docs, searches through random blogs and lots of Googling.

Links to other open issues

  • BK docs were kind of findable by typing "buildkite <thing>" into google, I saw that there was a still-open issue raised in 2016 for making docs searchable, but I wanted to +1 it from here too.
  • I found two issues about undocumented environment variables, and this is a problem that I faced.
  • I struggled with knowing if I should bother with the docs site (because it didn't have a lot of the answers I was looking for), or trying to navigate one of the many buildkite repos. It looks like I wasn't the first.

It'd be great if there was a guide/full recipe for a rails app (or node, python, whatever customers are using most). I really felt like I was on my own here, and it was one failure after the next鈥 I know that's the nature of test suites, but I think some improvements to the existing docs and general standardisation would vastly improve the experience. 馃憤


This comment has been minimized.

Copy link

commented May 25, 2018

This is amazing, thank you @benschwarz, we'll digest.


This comment has been minimized.

Copy link

commented May 25, 2018

Thanks for taking the time to document the experience! 馃檹馃徏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can鈥檛 perform that action at this time.