From 8e1cfaee95a61c5f8bf6b9142f08cf61f76f32a4 Mon Sep 17 00:00:00 2001 From: Sarah Jackson Date: Mon, 7 Jul 2025 12:40:29 +1000 Subject: [PATCH 1/4] Update Rails Parallel Example (consistency sweep) --- .buildkite/template.yml | 11 +++++-- License.md => LICENSE.md | 2 +- README.md | 64 ++++++++++++++++++++++++++++++++++++++++ Readme.md | 42 -------------------------- 4 files changed, 73 insertions(+), 46 deletions(-) rename License.md => LICENSE.md (96%) create mode 100644 README.md delete mode 100644 Readme.md diff --git a/.buildkite/template.yml b/.buildkite/template.yml index 37ac127..8234252 100644 --- a/.buildkite/template.yml +++ b/.buildkite/template.yml @@ -1,5 +1,10 @@ -# Used by the 'Add to Buildkite' button in the readme name: "Rails Parallel Example" +description: "An example pipeline that runs Rails tests in parallel across multiple agents using Knapsack." +emoji: ":rails:" +color: "#FDF5F5" +languages: + - "Ruby" + - "YAML" steps: - - label: ":pipeline:" - command: "buildkite-agent pipeline upload" + - command: "buildkite-agent pipeline upload" + label: ":pipeline:" diff --git a/License.md b/LICENSE.md similarity index 96% rename from License.md rename to LICENSE.md index f0a2db8..66db09b 100644 --- a/License.md +++ b/LICENSE.md @@ -1,6 +1,6 @@ The MIT License (MIT) -Copyright (c) 2016 Buildkite Pty Ltd +Copyright (c) 2025 Buildkite Pty Ltd Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md new file mode 100644 index 0000000..6e7cf41 --- /dev/null +++ b/README.md @@ -0,0 +1,64 @@ +# Buildkite Rails Parallel Pipeline Example + +[![Build status](https://badge.buildkite.com/aab023f2f33ab06766ed6236bc40caf0df1d9448e4f590d0ee.svg?branch=main)](https://buildkite.com/buildkite/rails-parallel-example/builds/latest?branch=main) +[![Add to Buildkite](https://img.shields.io/badge/Add%20to%20Buildkite-14CC80)](https://buildkite.com/new) + +This repository is an example [Buildkite](https://buildkite.com/) pipeline that demonstrates how to run 20 parallel test jobs for a [Rails](https://rubyonrails.org/) application using [rbenv](https://github.com/rbenv/rbenv) and [Knapsack](https://github.com/ArturT/knapsack). + +👉 **See this example in action:** [buildkite/rails-parallel-example](https://buildkite.com/buildkite/rails-parallel-example/builds/latest?branch=main) + +[![Add to Buildkite](https://buildkite.com/button.svg)](https://buildkite.com/new) + + + Screenshot of Buildkite Rails Parallel example pipeline + + + + +## How it works + +This repository doesn't include a Rails application, but it does provide the CI setup you'd typically use in a real project. + +In the [pipeline configuration file](.buildkite/pipeline.yml) the `parallelism` property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can then run 20 agents (on the same machine, or distributed) to work on the 20 jobs in parallel. + +This example: +- Uses rbenv to manage Ruby versions +- Runs tests in parallel using Buildkite’s `parallelism` feature +- Leverages [Knapsack](https://github.com/ArturT/knapsack) for test suite splitting +- Can be used on a single machine or distributed across multiple agents + +Key files: +- [.buildkite/hooks/environment](.buildkite/hooks/environment): loads rbenv before steps run +- [.buildkite/pipeline.yml](.buildkite/pipeline.yml): sets up parallel steps +- [scripts/ci/setup.sh](scripts/ci/setup.sh): prepares the CI environment +- [scripts/ci/parallel_specs.sh](scripts/ci/parallel_specs.sh): runs specs via Knapsack + +### Parallel test execution + +The test step in [`pipeline.yml`](.buildkite/pipeline.yml) is configured like this: + +```yml +steps: + - name: ":rspec:" + command: "scripts/ci/setup.sh && scripts/ci/parallel_specs.sh" + parallelism: 20 +``` + +Each parallel job receives unique environment variables (`BUILDKITE_PARALLEL_JOB`, `BUILDKITE_PARALLEL_JOB_COUNT`) to help split the workload. + +See the [parallelizing builds guide](https://buildkite.com/docs/guides/parallelizing-builds) for more information to create parallelized and distributed builds with Buildkite. + +> 💡 Want to use Docker? Try the [Rails Docker Parallel Example](https://github.com/buildkite/rails-docker-parallel-example) + +## Knapsack Pro + +Consult the [Knapsack documentation](https://github.com/ArturT/knapsack) for configuring your database and dependent services to support running parallel steps on the one machine. + +See [Knapsack Pro version of this example pipeline](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) for how to perform dynamic splits/allocation of test nodes. + +## License + +See [LICENSE.md](LICENSE.md) (MIT) + + + diff --git a/Readme.md b/Readme.md deleted file mode 100644 index 21183da..0000000 --- a/Readme.md +++ /dev/null @@ -1,42 +0,0 @@ -# Buildkite Rails Parallel Pipeline Example - -[![Build status](https://badge.buildkite.com/aab023f2f33ab06766ed6236bc40caf0df1d9448e4f590d0ee.svg?branch=main)](https://buildkite.com/buildkite/rails-parallel-example) -[![Add to Buildkite](https://buildkite.com/button.svg)](https://buildkite.com/new) - -This repository is an example on how to easily run 20 parallel testing agents for a [Rails](https://rubyonrails.org/) application using [Buildkite](https://buildkite.com/), [rbenv](https://github.com/rbenv/rbenv) and [Knapsack](https://github.com/ArturT/knapsack). - -See the full [Getting Started Guide](https://buildkite.com/docs/guides/getting-started) for step-by-step instructions on how to get this running, or [Add to Buildkite](https://buildkite.com/new) to try it yourself. - -Screenshot of Buildkite Rails Parallel example pipeline - -This repository doesn't actually include a Rails sample application, but has the same CI scripts you would use for a production application. - -Files to note: - -* [.buildkite/hooks/environment](.buildkite/hooks/environment) - loads rbenv -* [.buildkite/pipeline.yml](.buildkite/pipeline.yml) - the pipeline config showing how to have parallel steps -* [scripts/ci/setup.sh](scripts/ci/setup.sh) - sets up the CI environment for each step -* [scripts/ci/parallel_specs.sh](scripts/ci/parallel_specs.sh) - runs the specs in parallel using Knapsack - -## How does it work? - -In the [pipeline configuration file](.buildkite/pipeline.yml) the `parallelism` property for the test step is set to 20. When a build, the step will appear 20 times in the pipeline, each with different environment variables exposed so you can divvy up your test suite accordingly. You can then run 20 agents (on the same machine, or distributed) to work on the 20 jobs in parallel. - -```yml -steps: - - name: ":rspec:" - command: "scripts/ci/setup.sh && scripts/ci/parallel_specs.sh" - parallelism: 20 -``` - -See the [parallelizing builds guide](https://buildkite.com/docs/guides/parallelizing-builds) for more information to create parallelized and distributed builds with Buildkite. - -Consult the [Knapsack documentation](https://github.com/ArturT/knapsack) for configuring your database and dependent services to support running parallel steps on the one machine. - -## Knapsack Pro - -See [Knapsack Pro version of this example pipeline](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) for how to perform dynamic splits/allocation of test nodes. - -## License - -See [Licence.md](Licence.md) (MIT) From 5625c40266f613fd6736c129c422b3a9fb69f6b6 Mon Sep 17 00:00:00 2001 From: Sarah Jackson <78014112+123sarahj123@users.noreply.github.com> Date: Mon, 7 Jul 2025 14:43:21 +1200 Subject: [PATCH 2/4] Update README.md --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 6e7cf41..4b3aeab 100644 --- a/README.md +++ b/README.md @@ -56,6 +56,8 @@ Consult the [Knapsack documentation](https://github.com/ArturT/knapsack) for con See [Knapsack Pro version of this example pipeline](https://github.com/KnapsackPro/buildkite-rails-parallel-example-with-knapsack_pro) for how to perform dynamic splits/allocation of test nodes. + + ## License See [LICENSE.md](LICENSE.md) (MIT) From 5fe2d3bb6a5ccbfcbf7f10f0e1f52a103f24c60e Mon Sep 17 00:00:00 2001 From: Sarah Jackson <78014112+123sarahj123@users.noreply.github.com> Date: Mon, 7 Jul 2025 16:31:19 +1200 Subject: [PATCH 3/4] Update README.md remove whitespace --- README.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/README.md b/README.md index 4b3aeab..c320400 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,3 @@ See [Knapsack Pro version of this example pipeline](https://github.com/KnapsackP ## License See [LICENSE.md](LICENSE.md) (MIT) - - - From 85a1423292ae7d744e4f9b9b970b56b42580984d Mon Sep 17 00:00:00 2001 From: Sarah Jackson <78014112+123sarahj123@users.noreply.github.com> Date: Mon, 7 Jul 2025 16:42:49 +1200 Subject: [PATCH 4/4] Update .buildkite/template.yml --- .buildkite/template.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.buildkite/template.yml b/.buildkite/template.yml index 8234252..5b098e6 100644 --- a/.buildkite/template.yml +++ b/.buildkite/template.yml @@ -4,7 +4,7 @@ emoji: ":rails:" color: "#FDF5F5" languages: - "Ruby" - - "YAML" + - "Shell" steps: - command: "buildkite-agent pipeline upload" label: ":pipeline:"