Document how to run multiple agents with different config files

[toolmantim]

In the docs we show how to run multiple agents but we don't show any way to customise the agent configuration files for each agent… say, for giving them different tags/meta-data.

Someone just submitted the process it took for them:

To run multiple agents, on the same host, with different configurations, and behind a proxy server.

Once you have installed and tested an agent as per the installation instructions.

## Kill the single agent.

Stop the service and disable it "sudo systemctl stop buildkite-agent" and "sudo systemctl disable buildkite-agent"

## Config Files.

In the buildkite-agent home directory, `/ect/buildkite-agent` make a copy of buildkite-agent.config for each different config you want, e.g. `buildkite-agent-1.cfg`, `buildkite-agent-2.cfg`, etc. The name is irrelevant so call it what you like.

Then edit the new config files and set whatever configuration you want - I just set different queue names in the meta-data/tags.

## System launch file.

This file is `/lib/system/system/buildkite-agent.service`

Make a copy for each agent you want to run, i.e. `buildkite-agent@1.service`, `buildkite-agent@1.service`, etc., and edit this file.

There is an ExecStart line that launches the service, add to the end of this, in each file, ` -config /etc/buildkite-agent/buildkite-agent-1.cfg` or whichever conf you want to use.

Now you can enable the service, `sudo systemctl enable buildkite-agent@1`, and it should run.

## Repeat for however many agent you have setup.

This creates a symlink called `/etc/system/system/mutli-user.target.wants/buildkite-agent@1.service` that should link to the system launch file `/lib/sytemd/system/buildkite-agent@1.service` that was created earlier.

## Start the agents

Finally start the agents: `sudo systemctl start buildkite-agent@1`

[harrietgrace]

@toolmantim is this still the only way to have multiple config files? Does the config path env var/flag not allow you to specify different files?

[toolmantim]

I'm not exactly sure of the steps… I think there's a few different ways to do it. At a base level the agent let's you customise the config path, and create separate configuration files, but the Ubuntu-specific instructions here https://buildkite.com/docs/agent/v3/ubuntu#running-multiple-agents don't tell you how to do it — I believe it only shows how to run multiple agents from the same config file, because it's using a systemd template: https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files#creating-instance-units-from-template-unit-files

@sj26 is our go-to systemd expert!

[harrietgrace]

Looks like this is related to #186!

@sj26 is there a 'recommended' way to do separate config files with systemd templates?

[sj26]

Hi folks! 👋

The systemd way to do this is probably using a template instance specific drop-in file.

buildkite-agent@.service is a service template. buildkite-agent@custom.service is an instance with the instance name "custom".

buildkite-agent@custom.service inherits the config from the default at /lib/systemd/system/buildkite-agent@.service. We can add instance specific config in a drop-in (describe in the systemd.unit man page) named after the template instance at /etc/systemd/system/buildkite-agent@custom.service.d/custom-config.conf with contents like:

[Service]
Environment=BUILDKITE_AGENT_CONFIG=/etc/buildkite-agent/buildkite-agent-custom.cfg

This will override the config location for this one particular service instance.

You can verify this is working using systemctl cat ...:

root@468ee23db373:/# systemctl cat buildkite-agent@custom.service
# /lib/systemd/system/buildkite-agent@.service
[Unit]
Description=Buildkite Agent (%i)
Documentation=https://buildkite.com/agent
After=syslog.target
After=network.target

[Service]
Type=simple
User=buildkite-agent
Environment=HOME=/var/lib/buildkite-agent
ExecStart=/usr/bin/buildkite-agent start
RestartSec=5
Restart=on-failure
TimeoutStartSec=10
TimeoutStopSec=5min
KillMode=mixed

[Install]
WantedBy=multi-user.target
DefaultInstance=1

# /etc/systemd/system/buildkite-agent@custom.service.d/custom-config.conf
[Service]
Environment=BUILDKITE_AGENT_CONFIG=/etc/buildkite-agent/buildkite-agent-custom.cfg

[toolmantim]

It'd be great to figure out a real life example, so we could use that instead of "custom". If you're doing a custom agent config, it's probably also on a separate queue? Or perhaps a separate organization?

Would you also probably want to override the unit’s Description=Buildkite Agent (%i) value too?

[sj26]

What about a scenario where you want to run a few agents per host, plus an agent dedicated to pipeline uploads on a separate queue?

$ systemctl edit buildkite-agent@pipeline-uploader.service
# opens /etc/systemd/system/buildkite-agent@pipeline-uploader.service.d/override.conf in $EDITOR

[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader

$ systemctl cat buildkite-agent@pipeline-uploader.service
# /lib/systemd/system/buildkite-agent@.service
[Unit]
Description=Buildkite Agent (%i)
Documentation=https://buildkite.com/agent
After=syslog.target
After=network.target

[Service]
Type=simple
User=buildkite-agent
Environment=HOME=/var/lib/buildkite-agent
ExecStart=/usr/bin/buildkite-agent start
RestartSec=5
Restart=on-failure
TimeoutStartSec=10
TimeoutStopSec=5min
KillMode=mixed

[Install]
WantedBy=multi-user.target
DefaultInstance=1

# /etc/systemd/system/buildkite-agent@pipeline-uploader.service.d/override.conf
[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader

$ systemctl enable --now buildkite-agent@pipeline-uploader
Created symlink from /etc/systemd/system/multi-user.target.wants/buildkite-agent@pipeline-uploader.service to /lib/systemd/system/buildkite-agent@.service.

$ systemctl enable --now buildkite-agent@1
Created symlink from /etc/systemd/system/multi-user.target.wants/buildkite-agent@1.service to /lib/systemd/system/buildkite-agent@.service.

$ systemctl enable --now buildkite-agent@2
Created symlink from /etc/systemd/system/multi-user.target.wants/buildkite-agent@2.service to /lib/systemd/system/buildkite-agent@.service.

$ systemctl enable --now buildkite-agent@3
Created symlink from /etc/systemd/system/multi-user.target.wants/buildkite-agent@3.service to /lib/systemd/system/buildkite-agent@.service.

$ systemctl enable --now buildkite-agent@4
Created symlink from /etc/systemd/system/multi-user.target.wants/buildkite-agent@4.service to /lib/systemd/system/buildkite-agent@.service.

The default systemd description makes sure the instance name is interpolated, so it looks nice:

[hofstee]

What if I wanted to make let's say 5 copies of a custom service? I could use BUILDKITE_AGENT_SPAWN but then they all have the same name and can't run at the same time because they all want to work from the same directory. I can't seem to do buildkite-agent@custom.1 or anything of the sort because it just launches the default buildkite-agent@.service. Do I have to make N custom services? Can I somehow get the names to be decorated by the spawn count?

[sj26]

@hofstee ah, in that case you might like to add %n to the agent name which will be substituted by a number which ensures the agent name is unique across all your agents (docs). So for a custom service drop-in, it's as above but with:

[Service]
Environment=BUILDKITE_AGENT_NAME=%%hostname-pipeline-uploader-%%n
Environment=BUILDKITE_AGENT_TAGS=queue=pipeline-uploader
Environment=BUILDKITE_AGENT_SPAWN=5

You could probably also use systemd's %i and friends.

[sj26]

(The default buildkite agent configuration uses the name %hostname-%n.)