[Rust] [CI] Try out buildkite

[asfimport]

Let's make a test to validate whether we can use buildkite on our own flows, which adds a lot of options in architectures and environments that we can test stuff on.

Goal: validate that we can use buildkite on the rust builds.

Requirements:

1. pipeline starts when a PR is made

2. result is sent back to github and users can access its logs

3. we can use caches (e.g. https://github.com/danthorpe/cache-buildkite-plugin )

4. we can actually run the builds

5. we can limit the builds to only be triggered when certain parts of the repo change (i.e. not run when only C++ code changed)

    

Reporter: Jorge Leitão / @jorgecarleitao

PRs and other links:

• GitHub Pull Request #9111

_{Note: This issue was originally created as ARROW-11140. Please see the migration documentation for further details.}

[asfimport]

Jorge Leitão / @jorgecarleitao:
[~wesm_impala_7e40], would it be possible to add a new pipe with a single command,

buildkite-agent pipeline upload .buildkite/rust.yml

with the default github integration?

[asfimport]

Jorge Leitão / @jorgecarleitao:
The main blocker I see atm is that buildkite does not support starting pipelines only if some part of the code changes: buildkite/feedback#243

[asfimport]

Jorge Leitão / @jorgecarleitao:
Example of the pipeline running on a mac: https://buildkite.com/test-811/example/builds/36#c19196dc-72dd-4c3c-868e-893dd392a63a

can't make cache working. can't make artifacts working.

[asfimport]

Antoine Pitrou / @pitrou:
You can still adapt the ci/detect_changes.py script to limit certain builds depending on the changed files.

[asfimport]

Jorge Leitão / @jorgecarleitao:
@pitrou, thanks! @kszucs solved this in a smarter way within buildkite. I was also able to address the caching / artifacts, which unblocks this.

I have pushed the latest version to github. @kszucs, it assumes that there are 3 queues:

• docker, any machine that can run a docker job using the docker plugin (basically can run docker)

• macos, macos machine with rust installed

• windows, windows machine with rust installed

  If / when we have the runners, the pipeline should reproduce the same jobs that we have on github for Rust.

  I do not have strong opinions here, I just used the same setup that we have on github wrt to the runners. It looks as follows on my repo / account, where I have one (macos) agent: https://buildkite.com/test-811/example/builds?branch=buildkite

   

[asfimport]

Jorge Leitão / @jorgecarleitao:
I cannot proceed further without the runners on apache account and therefore will park for now. Thanks a lot for your help!

[asfimport]

Andrew Lamb / @alamb:
Migrated to github: apache/arrow-rs#165