Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
180 lines (117 sloc) 7.75 KB
# Defining Your Pipeline Steps
Pipeline steps can be defined visually in Buildkite or, more powerfully, in code using a `pipeline.yml` file.
Defining your pipeline steps in a `pipeline.yml` file gives you access to more configuration options than the web interface, and allows you to version, audit and review your build pipelines alongside your source code.
<%= toc %>
## Getting started
After creating a [pipeline](/docs/pipelines), you can add steps in the Buildkite Pipeline Settings. In the 'Steps' section, click the ➕ to add a step. You can create your entire pipeline by adding steps in this way.
To use a pipeline.yml file, click the ➕ to add a step and select “Read steps from repository”:
<%= image "pipeline-upload-step.png", width: 612/2, height: 484/2, alt: "Screenshot of the 'Read steps from repository' menu item" %>
Your pipeline will now show a new step labeled :pipeline: that runs a `buildkite-agent pipeline upload` command:
<%= image "pipeline-upload.png", width: 1174/2, height: 452/2, alt: "Screenshot of the pipeline upload step" %>
When this step runs on an agent, it adds the steps you defined in `.buildkite/pipeline.yml` to your pipeline. For example, here is a pipeline file with a single step:
```yml
steps:
- label: Example Test
command: echo "Hello!"
```
The above example is a [command step](/docs/pipelines/command-step), which will run the code `echo "Hello!"` and print "Hello!" to the build's log.
To test your `.buildkite/pipeline.yml` file, commit your `pipeline.yml` file and push it to your repository. If you've set up webhooks, this will automatically create a new build. You can also create a new build using the 'New Build' button on the pipeline page.
<%= image "show-example-test.png", width: 968/2, height: 698/2, alt: "Screenshot of the build passing with pipeline upload step first, and then the example step" %>
If your pipeline has a lot of steps and you have multiple agents available to run them, they will automatically run at the same time. If your steps rely on running in sequence, you can use [wait steps](/docs/pipelines/wait-step) to ensure that the steps before the 'wait' are completed before subsequent steps are run.
When each step is run by an agent, it will be run with a clean checkout of the pipeline's repository. If your commands or scripts rely on the output from previous steps, you will need to either combine them into a single script or use artifacts to pass data between steps. This enables any step to be picked up by any agent and run steps concurrently to speed up your build.
## Example pipeline
Here’s a more complete example based on [our Buildkite Agent’s build pipeline](https://github.com/buildkite/agent/blob/master/.buildkite/pipeline.yml). It contains script commands, wait steps, block steps, and automatic artifact uploading:
```yaml
steps:
- label: '\:hammer\: Tests'
command: 'scripts/tests.sh'
env:
BUILDKITE_DOCKER_COMPOSE_CONTAINER: app
- wait
- label: '\:package\: Package'
command: 'scripts/build-binaries.sh'
artifact_paths: 'pkg/*'
env:
BUILDKITE_DOCKER_COMPOSE_CONTAINER: app
- wait
- label: '\:debian\: Publish'
command: 'scripts/build-debian-packages.sh'
artifact_paths: 'deb/**/*'
branches: 'master'
agents:
queue: 'deploy'
- block: '\:shipit\: Release'
branches: 'master'
- label: '\:github\: Release'
command: 'scripts/build-github-release.sh'
artifact_paths: 'releases/**/*'
branches: 'master'
- wait
- label: '\:whale\: Update images'
command: 'scripts/release-docker.sh'
branches: 'master'
agents:
queue: 'deploy'
```
## Step types
There are four different step types you can use in your Buildkite pipelines. Click through each step type for detailed usage documentation:
* [Command Step](/docs/pipelines/command-step)
* [Wait Step](/docs/pipelines/wait-step)
* [Block Step](/docs/pipelines/block-step)
* [Trigger Step](/docs/pipelines/trigger-step)
## Customizing the pipeline upload path
By default the pipeline upload step reads your pipeline definition from `.buildkite/pipeline.yml` in your repository. You can specify a different file path by adding it as the first argument:
<%= image "custom-pipeline-file.png", width: 1174/2, height: 452/2, alt: "Screenshot of a custom pipeline file upload" %>
A common use for custom file paths is when separating test and deployment steps into two completely separate pipelines, both using the same repository URL. For example, your test pipeline’s upload command could be:
```
buildkite-agent pipeline upload .buildkite/pipeline.yml
```
And your deployment pipeline’s upload command could be:
```
buildkite-agent pipeline upload .buildkite/pipeline.deploy.yml
```
For a list of all command line options, see the [buildkite-agent pipeline upload](/docs/agent/v3/cli-pipeline#uploading-pipelines) documentation.
## Dynamic pipelines
Because the pipeline upload step runs on your agent machine, you can generate pipelines dynamically using scripts from your source code. This provides you with the flexiblity to structure your pipelines as you wish.
The following example generates a list of parallel test steps based upon the `test/*` directories within your repository:
```
#!/bin/bash
# exit immediately on failure, or if an undefined variable is used
set -eu
# begin the pipeline.yml file
echo "steps:"
# add a new command step to run the tests in each test directory
for test_dir in test/*/; do
echo " - command: \"run_tests "${test_dir}"\"
done
```
To use this script, you'd save it to `.buildkite/pipeline.sh` inside your repository, ensure it is executable, and then update your pipeline upload step to use the new script:
```bash
.buildkite/pipeline.sh | buildkite-agent pipeline upload
```
When the build is run it will execute the script and pipe the output to the `pipeline upload` command. The upload command will insert the steps from the script into the build immediately after the upload step.
In the below `pipeline.yaml` example, when the build runs it will execute the `.buildkite/pipeline.sh` script, then the test steps from the script will be added to the build before the wait step and command step. After the test steps have run, the wait and command step will run.
```bash
steps:
- command: .buildkite/pipeline.sh | buildkite-agent pipeline upload
label: \:pipeline\: Upload
- wait
- command: "other-script.sh"
label: Run other operations
```
## Migrating existing steps to a pipeline.yml
If you have a pipeline that is defined in the web interface and would like to migrate it to a `pipeline.yml` without any interruption to builds, add a command step and wait step at the start of your pipeline:
For agents running Linux, macOS or other Unixes, the command is:
```bash
[[ -f ".buildkite/pipeline.yml" ]] && buildkite-agent pipeline upload --replace || true
```
Or, if your Agent runs Windows:
```bash
CMD /k if exist ".buildkite/pipeline.yml" buildkite-agent pipeline upload .buildkite/pipeline.yml --replace
```
This line of code will check if the pipeline file exists and, if it does, execute the `buildkite agent pipeline upload --replace` command to replace any steps that were defined via the web interface with those defined in the `pipeline.yml`.
To ensure that your existing steps don't run alongside the check, we include a [wait step](/docs/pipelines/wait-step) after it.
Your steps should then look something like this:
<%= image "pipeline-upload-migrate-existing-steps.png", width: 998/2, height: 940/2, alt: "Screenshot of the above described pipeline upload command, before some existing steps" %>
## Further documentation
You can also create pipelines using the `buildkite-agent` command line tool. See the [buildkite-agent pipeline documentation](/docs/agent/v3/cli-pipeline) for a full list of the available parameters.