Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Try out buildkite #165

Closed
alamb opened this issue Apr 26, 2021 · 2 comments
Closed

[CI] Try out buildkite #165

alamb opened this issue Apr 26, 2021 · 2 comments
Labels
arrow Changes to the arrow crate

Comments

@alamb
Copy link
Contributor

alamb commented Apr 26, 2021

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11140

Let's make a test to validate whether we can use buildkite on our own flows, which adds a lot of options in architectures and environments that we can test stuff on.

Goal: validate that we can use buildkite on the rust builds.

Requirements:

pipeline starts when a PR is made

result is sent back to github and users can access its logs

we can use caches (e.g. [https://github.com/danthorpe/cache-buildkite-plugin] )

we can actually run the builds

we can limit the builds to only be triggered when certain parts of the repo change (i.e. not run when only C++ code changed)

 

@alamb alamb added the arrow Changes to the arrow crate label Apr 26, 2021
@alamb
Copy link
Contributor Author

alamb commented Apr 26, 2021

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:07:55.582+0000:

[~wesm_impala_7e40], would it be possible to add a new pipe with a single command,

{{buildkite-agent pipeline upload .buildkite/rust.yml}}

with the default github integration?

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:16:00.080+0000:

The main blocker I see atm is that buildkite does not support starting pipelines only if some part of the code changes: [https://github.com/buildkite/feedback/issues/243]

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T06:26:09.316+0000:

Example of the pipeline running on a mac: [https://buildkite.com/test-811/example/builds/36#c19196dc-72dd-4c3c-868e-893dd392a63a]

can't make cache working. can't make artifacts working.

Comment from Antoine Pitrou(apitrou) @ 2021-01-06T09:39:03.248+0000:

You can still adapt the {{ci/detect_changes.py}} script to limit certain builds depending on the changed files.

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T19:17:45.515+0000:

[~apitrou], thanks! [~kszucs] solved this in a smarter way within buildkite. I was also able to address the caching / artifacts, which unblocks this.

I have pushed the latest version to github. [~kszucs], it assumes that there are 3 queues:
 * {{docker}}, any machine that can run a docker job using the docker plugin (basically can run docker)
 * {{macos}}, macos machine with rust installed
 * {{windows}}, windows machine with rust installed

If / when we have the runners, the pipeline should reproduce the same jobs that we have on github for Rust.

I do not have strong opinions here, I just used the same setup that we have on github wrt to the runners. It looks as follows on my repo / account, where I have one (macos) agent: [https://buildkite.com/test-811/example/builds?branch=buildkite]

 

Comment from Jorge Leitão(jorgecarleitao) @ 2021-01-06T19:20:05.869+0000:

I cannot proceed further without the runners on apache account and therefore will park for now. Thanks a lot for your help!

@jorgecarleitao
Copy link
Member

won't fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate
Projects
None yet
Development

No branches or pull requests

2 participants