Bisecting errors against PRs

[nikomatsakis]

One of the best way of figuring out the cause of a regression is bisecting against the PRs that have landed recently. There are a number of folks (cc'd below) who do this from time to time -- I'm not really sure what wizardry they use to do it, but I'd love to see it documented!

cc @Mark-Simulacrum, @kennytm, @TimNN, @est31

[nikomatsakis]

(Is this on topic for this guide? Maybe too far astray? Maybe better for forge? I'm not really sure.)

[Mark-Simulacrum]

The current (possibly broken, I haven't used it in a while) tooling that I know of to do this is https://github.com/Mark-Simulacrum/bisect-rust. However, I have had some thoughts about changing to just using normal git bisect in a rust-lang/rust checkout but instead of building downloading CI artifacts -- this should make the implementation considerably smaller. I hope to investigate something like this over the next few weeks, or maybe at all hands. I'd like for us to have an easy-to-use method that does bisection, and I don't think bisect-rust quite does that today.

[mark-i-m]

I think this could go in the "debugging the compiler" section

[est31]

Usually it looks like this for me:

1. reproduce the regression by using rustup to check two nightly versions around the regression date
2. use those two nightlies as boundaries for a bisect-rust search

[nikomatsakis]

While we're talking about tools, it'd be nice for the same tool to support nightlies too (and, in an ideal world, building from source).

[est31]

I've never built from source in my bisection work. I've always pinged the most suspicious looking person and then left it to them to nail stuff down further :). But yeah support for nightlies as well would be great (as we delete older builds now)

[mark-i-m]

@est31 Would you mind adding this to the "Debugging the compiler" chapter?

[ehuss]

I've written a tutorial on how to bisect PR's here: https://github.com/rust-lang-nursery/cargo-bisect-rustc/blob/master/TUTORIAL.md

[mark-i-m]

@ehuss Great :) Are you looking to move it to the book?

[ehuss]

That's up to @Mark-Simulacrum. I would think it will be easier to maintain in its own repo, in case things change. Perhaps a small paragraph directing people to the tool would be sufficient? Maybe something like this in the debugging chapter:

Narrowing Regressions

The cargo-bisect-rustc tool can be used as a quick and easy way to find exactly which PR caused a change in rustc behavior. It automatically downloads rustc PR artifacts and tests them against a project you provide until it finds the regression. You can then look at the PR to get more context on why it was changed. See the tutorial on how to use it.

[Mark-Simulacrum]

I'm happy with anything! No Rust team currently maintains cargo-bisect-rustc and I myself only have time to review pull requests currently but I think we can link to it despite both of these.

Generally linking the tutorial is I think a better approach so that it stays more up to date, though.

[mark-i-m]

opened #216

let me know what you think