Make ci-bench more deterministic #1444
Conversation
|
I have another improvement in mind, which is to automatically output the cachegrind diff for the benchmarks that have significant instruction differences. I'll leave that for tomorrow, though |
ci-bench/src/main.rs
Outdated
| @@ -116,8 +117,8 @@ fn main() -> anyhow::Result<()> { | |||
| .get(index as usize) | |||
| .ok_or(anyhow::anyhow!("Benchmark not found: {index}"))?; | |||
|
|
|||
| let mut stdin = io::stdin().lock(); | |||
| let mut stdout = io::stdout().lock(); | |||
| let mut stdin = util::UnbufferedStdinReader; | |||
There was a problem hiding this comment.
if i've understood this correctly, it would be equivalent to do let mut stdin = unsafe { File::from_raw_fd(0) };
There was a problem hiding this comment.
I like that! The only drawback I see is that each File will be closed upon drop, meaning that stdin and stdout will get closed too. I think that's not a real problem, though, because if we ever want to use stdio elsewhere, we'll want to do so in an unbuffered way (reusing the files).
(Just pushed the changes use File)
There was a problem hiding this comment.
Could do a mem::forget() for the files, I suppose?
There was a problem hiding this comment.
True, I'll add that for ultimate correctness
aochagavia
force-pushed
the
ci-bench-determinism
branch
2 times, most recently
from
1691a73
to
78a8b31
Compare
Codecov Report
@@ Coverage Diff @@
## main #1444 +/- ##
=======================================
Coverage 96.49% 96.49%
=======================================
Files 71 71
Lines 15160 15160
=======================================
Hits 14628 14628
Misses 532 532 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
aochagavia
force-pushed
the
ci-bench-determinism
branch
from
78a8b31
to
8370f0c
Compare
|
The failing CI jobs seem transient |
aochagavia
force-pushed
the
ci-bench-determinism
branch
from
8370f0c
to
c8be5c8
Compare
* Switch to unbuffered stdio * Use FxHasher where possible
Head branch was pushed to by a user without write access
aochagavia
force-pushed
the
ci-bench-determinism
branch
from
c8be5c8
to
d440dbc
Compare
Note: the CI failure is expected because the benchmarking code has changed between main and this branch (and because main is noisy).