CI benchmarks: provide a way to generate wall-time measurements

[aochagavia]

Our current benchmarking setup measures executed CPU instructions, which according to this blog post can reliably identify the speed difference between two versions of rustls:

It correlates fairly well with wall-time. You’d be mad to use it to compare the speed of two different programs, but it is very useful for comparing two slightly different versions of the same program, which are likely to have a similar instruction mix.

IMO we can safely assume most PRs propose changes that are "slightly different" from the existing version of the code, meaning that the instruction count is a reliable metric to judge their performance impact.

What about bigger changes? Talking to @nnethercote (author of the quote above), he mentioned that the correlation between instruction counts and wall-time decreases when there are bigger changes, or when they significantly affect memory layout. It will probably take some time to develop an intuition of which changes fall under this category (maybe #1448 is an instance), but it is clear that we need a setup to obtain wall-time measurements in those cases.

I'm currently thinking of adding flags to the bench runner that allow:

1. Measuring wall-time instead of icounts (running each benchmark multiple times);
2. Comparing the resulting time distributions between two runs.

The idea would be to manually trigger a wall-time bench run when a reviewer considers it necessary.

By the way, this all assumes we can run them on dedicated, properly configured, hardware. I'm currently arranging an OVH bare-metal machine sponsored by ISRG (let me know if you'd prefer something else).

[aochagavia]

We are close to deploying our bare-metal benchmark runner, with support for wall-time benchmarks. It is unclear to me, however, which scenarios we'd like to test in our wall-time benchmarks. Below are my current thoughts. Please have a look and tell me what you think ;)

Intuitively, it seems nice to have a wall-time benchmark counterpart for each icount benchmark, so we can see the wall-time change that corresponds to a reported icount change. There are some obstacles to that, however:

1. It is non-trivial to reuse the code from the existing icount benchmarks for wall-time benchmarking purposes. The icount code runs the server and the client parts of the connection as two separate processes and uses a stdio-based blocking communication channel between them. The wall-time benchmarks, on the other hand, need to run in a single process and thread to minimize noise, making it impossible to communicate using blocking code.
2. Given that wall-time benchmarks take longer to run (around 10 seconds each if we use criterion), we probably want to avoid having too many, to keep feedback loops tight.

With this in mind, I see the following alternatives:

1. If we want to have wall-time measurements for our existing benchmarks at all costs, we might be able to reach that goal by rewriting the existing benchmarks in async / await style and running them on custom executors (a blocking one, for the icount benchmarks; a non-blocking one, for the wall-time benchmarks). We can also mitigate the slowness of criterion benchmarks by running only those for the top 5 icount regressions (or something like that).
2. A variant of the alternative above would be to duplicate the benchmarking code instead of rewriting it in async / await style. This would involve between 200 and 400 LOC, which seems manageable. We'd need to be careful to keep everything in sync if there is ever a need to make changes.
3. Someone who knows the innards of the library could come up with 5 to 15 benchmark scenarios which exercise the most important code paths, in the hope they will reveal any relevant regressions.

I'm very interested in hearing what you, as maintainers, would like to have when reviewing PRs. Which of these options would help you most when you see an icount regression and want to know how serious it is? Or would you suggest a totally different approach?

[ctz]

How about starting with cargo run --release --example bench and going from there? I have found that very noisy in the past, but I've put that down to having none of the "properly configured" items done locally, and running it alongside a bunch of other processes (browsers, spotify, mail etc etc). So I'm keen to see if the quieter environment makes those numbers more useful.

[epompeii]

2. A variant of the alternative above would be to duplicate the benchmarking code instead of rewriting it in async / await style. This would involve between 200 and 400 LOC, which seems manageable. We'd need to be careful to keep everything in sync if there is ever a need to make changes.

How hard would it be to wrap this in a macro_rules macro and just generate the two versions required? Keep things DRY.

This may be overkill but a possible option 4. Create a custom benchmark harness that handles the multiplexing. The same benchmark could be run in wall-time mode or instruction count mode, depending on how the harness is invoked.

[cpu]

The wall-time benchmarks, on the other hand, need to run in a single process and thread to minimize noise, making it impossible to communicate using blocking code.

Forgive my ignorance, is this from the unpredictability of thread scheduling? How noisy does it end up being on the dedicated hardware? 🤔

How about starting with cargo run --release --example bench and going from there?

I like that idea. I think the other options are heavier weight and this could be tried without as much up-front investment. If it turns out to be unworkable we could revisit the more involved options.

[aochagavia]

Forgive my ignorance, is this from the unpredictability of thread scheduling? How noisy does it end up being on the dedicated hardware? 🤔

I'm not an expert either (and haven't compared dedicated vs other hardware), but the combination between multi-threading and blocking waits sounds like it could indeed introduce noise because of thread scheduling. IMO keeping things single-threaded is a more promising avenue to explore.

How about starting with cargo run --release --example bench and going from there?

I'll give it a try! Hope we can get away with it!

How hard would it be to wrap this in a macro_rules macro and just generate the two versions required?

My experience with macros is painful enough that I'd rather give async / await a try first (it'd probably be more readable). In any case, it looks like we will be able to sidestep the dilemma for the time being by reusing the existing benchmarks in the repository (cargo run --release --example bench)

[aochagavia]

All right, I have a branch of the bench runner that runs cargo run --release --example bench. See this comment for the resulting comparison.

Here are a few bits of extra context:

• I configured a noise threshold of 5% when there are not enough historical results to automatically derive it.
• Since we are just experimenting, the bench runner goes through all 96 combinations of parameters, which results in 192 benchmark scenarios (client and server results are reported separately).

With this in mind, here are some steps we could take:

1. Create a list of parameters that are interesting to benchmark and ignore the rest. We can then run the cargo run --release --example bench for each parameter set. Does anyone have suggestions on which parameters are most relevant (TLS version, key kind, cipher suite, etc)?
2. Run cargo run --release --example bench multiple times, so we can reduce noise through statistics (the linked comment shows noise below 5%, but a later run reached 8.9%, which IMO means we can't rely on a single bench run).

[aochagavia]

Today I've been doing more rigorous experimentation. Specifically, I ran cargo run --release --example bench 30 times and calculated the noise threshold based on the results (i.e. treating them as historic data). I'm using 1% as the minimum threshold.

Results:

• About half of the benchmarks have noise thresholds above 5%.
• All 30 runs had one or more false positives.
• The highest number of false positives for a run was 9. The median was 4.

Also interesting:

• Raising the minimum threshold to 2% gets us a lower amount of false positives per run, but still 29 out of 30 runs have false positives. The median is 2 per run.
• Running the process in a dedicated CPU core (using isolcpus) doesn't seem to make a difference (30 runs yielded similarly noisy results).
• The results after repeating the experiment (another 30 runs) suggest that the noise is not localized to a specific set of benchmarks.

Based on the above results, my impression is that --example bench is too unreliable to be useful when reviewing PRs. I'll be experimenting with Criterion to see if it gives us more reliable results. Also, if anyone has additional suggestions, please chime in!

[epompeii]

One approach I have heard is pretty effective is running benchmarks a few times, say three, and then taking the minimum value and using that for comparisons. This paper is on my to read list: https://arxiv.org/pdf/1608.04295.pdf
I'm curious if that would help with the false positive rate here.

[aochagavia]

One approach I have heard is pretty effective is running benchmarks a few times, say three, and then taking the minimum value and using that for comparisons.

I considered that option, but it unfortunately comes with its own drawbacks (see, for instance, this article). If possible, I'd like to reach a solution that takes the distribution of the results into account (like criterion).

[ctz]

Specifically, I ran cargo run --release --example bench 30 times and calculated the noise threshold based on the results (i.e. treating them as historic data).

Is this for a specific test case, or the whole time the process takes?

In any case this kind of corroborates my experience of trying to compare the result from these benchmarks. I will look into this further.

[aochagavia]

Each run went through all 96 test cases (each case producing 2 throughput measurements, either MB/s or handshakes/s). From the output, I obtained the "result history" and fed it into the noise threshold calculator. You can think of it as outputting the results of each cargo run --release --example bench to a file, loading all files and grouping the results by line number.

Does this answer the question or were you referring to something else?