Skip to content

Commit

Permalink
pkg/bisect: detect when too many instances errored
Browse files Browse the repository at this point in the history
We currently skip a commit iff all 10 instances errored.
But if, say, only 9 errored we consider it as OK,
but this significnalty reduces chances of detecting flaky crashes.
So skip if more than 2/3 errored.

Update #501
  • Loading branch information
dvyukov committed Mar 22, 2019
1 parent 9ad9ef2 commit 028c095
Showing 1 changed file with 6 additions and 1 deletion.
7 changes: 6 additions & 1 deletion pkg/bisect/bisect.go
Expand Up @@ -275,7 +275,8 @@ func (env *env) test() (vcs.BisectResult, *vcs.Commit, *report.Report, error) {
return vcs.BisectSkip, current, nil, nil
}
testStart := time.Now()
results, err := env.inst.Test(10, cfg.Repro.Syz, cfg.Repro.Opts, cfg.Repro.C)
const numTests = 10
results, err := env.inst.Test(numTests, cfg.Repro.Syz, cfg.Repro.Opts, cfg.Repro.C)
env.testTime += time.Since(testStart)
if err != nil {
env.log("failed: %v", err)
Expand All @@ -285,6 +286,10 @@ func (env *env) test() (vcs.BisectResult, *vcs.Commit, *report.Report, error) {
res := vcs.BisectSkip
if bad != 0 {
res = vcs.BisectBad
} else if numTests-good-bad > numTests/3*2 {
// More than 2/3 of instances failed with infrastructure error,
// can't reliably tell that the commit is good.
res = vcs.BisectSkip
} else if good != 0 {
res = vcs.BisectGood
}
Expand Down

0 comments on commit 028c095

Please sign in to comment.