Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust/Cargo CI perf improvements #6119

Closed
jsirois opened this issue Jul 13, 2018 · 3 comments
Closed

Rust/Cargo CI perf improvements #6119

jsirois opened this issue Jul 13, 2018 · 3 comments
Assignees

Comments

@jsirois
Copy link
Contributor

jsirois commented Jul 13, 2018

A few ideas stumbled upon while trying to speed up CI burns on #6104:

  • Cache src/rust/engine/target in .travis.yml
    The official rust language support in Travis caches ~/.cargo and target/, so this is kosher from their point of view and blessed by Chrichton. I tried this already and the upshot is all 3rdparty crate compilations are cached when they have not changed versions - easy win.
  • Throttle cargo build as we do for zinc in pants.travis-ci.ini
    This can be done via CARGO_BUILD_JOBS=N in .travis.yml and might give us better throughput with less thrash. Measurements needed.
  • Throttle cargo test similar to above.
    Not sure how to best do this. There is rumor RUST_TEST_THREADS=N works, though it looks discouraged. I've had great luck with cargo make ... which allows setting up arbitrary commands in a Makefile.toml on a side project. Might be a way to go for these bottom two bullets. Likely very little bang for the buck here though, tests are still pretty fast.
@illicitonion
Copy link
Contributor

We already throttle cargo test for OSX:

if [[ "$(uname)" == "Darwin" ]]; then
# The osx travis environment has a low file descriptors ulimit, so we avoid running too many
# tests in parallel.
test_threads_flag="--test-threads=1"
fi

easy to extend to Linux if needed (but realistically, that shard runs pretty fast)

@jsirois jsirois self-assigned this Jul 21, 2018
@jsirois
Copy link
Contributor Author

jsirois commented Jul 21, 2018

Hitting the 1st bullet in #6208

jsirois added a commit to jsirois/pants that referenced this issue Aug 6, 2018
Picks the lowest hanging fruit from pantsbuild#6119.
jsirois added a commit to jsirois/pants that referenced this issue Aug 6, 2018
Picks the lowest hanging fruit from pantsbuild#6119.
jsirois added a commit that referenced this issue Aug 7, 2018
Picks the lowest hanging fruit from #6119.
@jsirois
Copy link
Contributor Author

jsirois commented Aug 7, 2018

Alright, #6309 cuts ~4 minutes per shard. I'm going to stop progress here and close.

@jsirois jsirois closed this as completed Aug 7, 2018
CMLivingston pushed a commit to CMLivingston/pants that referenced this issue Aug 27, 2018
Picks the lowest hanging fruit from pantsbuild#6119.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants