Performance Testing

[markmandel]

We should have some kind of repeatable performance harness, ideally running on a regular basis, so we can see if have performance problems or regressions.

[markmandel]

Suggestion: Stick it in front of a game and see what happens.

Could use Xonotic or Supertuxkart from Agones as an example (in fact, could be an interesting example to have on hand anyway).

[markmandel]

Working on an example with https://iperf.fr/iperf-doc.php#3doc to demo sending lots of packets through - this can be then turned into a proper perf test.

[markmandel]

About to submit a PR, so that everyone can replicate locally, but wanted to capture this.

When running iperf3 against itself on my local machine (no quilkin proxy), I get this result:

[SUM]   0.00-60.00  sec  2.10 GBytes   300 Mbits/sec  0.000 ms  0/68700 (0%)  sender
[SUM]   0.00-60.00  sec  2.10 GBytes   300 Mbits/sec  0.009 ms  0/68700 (0%)  receiver

Very little jitter, and nothing dropped.

Unfortunately, when I do the same test with Quilkin in between I see the the following:

[SUM]   0.00-60.00  sec  2.10 GBytes   300 Mbits/sec  0.000 ms  0/68700 (0%)  sender
[SUM]   0.00-60.00  sec  1.40 GBytes   200 Mbits/sec  0.288 ms  22635/68521 (33%)  receiver

As you can see, quite a bit of packet loss and some pretty nasty jitter. I've run this a few times and seen similar results.

I'm not 100% sure it's Quilkin just because I had to do some TCP tunnelling to get iperf3 to work with Quilkin between, and I'm assuming I got it all correct, but worth noting.

I capture Quilkin's prometheus metrics, along with all the logs, and they list no packets dropped, or errors. iperf3.zip

@iffyio when you did you load tests, did you get reports back on packets dropped?

The good new is - CPU is low for the number of packets I'm sending, and memory is very stable.

[iffyio]

Packet loss would be expected if quilkin isn't keeping up with what iperf3 is sending - its recv buffer gets full and the os drops any new packets, which would be fine. As long as quilkin is forwarding all packets it receives which sounds like thats the case. In my previous tests I didn't explicitly check for packet loss but I was pretty sure some where being dropped since the sender there sent as fast as was possible

[markmandel]

Another idea benchmarking: https://www.jibbow.com/posts/criterion-flamegraphs/

In general https://github.com/tikv/pprof-rs looks pretty awesome for looking into things. Shame it doesn't do heap analysis as well.

[markmandel]

Packet loss would be expected if quilkin isn't keeping up with what iperf3 is sending

🤔 CPU is definitely not a bottleneck here. But the point of setting this up is so we can dig into these type of issues. We can keep building out tooling and see where we end up.

I'll adjust the PR to be something that passes a bit better, rather than an overload, and we can take things from there.

[markmandel]

Dropping the parallelism to 75 (on my laptop at least) means that packets don't drop -- so it does seem like an overwhelming of the system issue.

I am also realising that i am not smart, and I'm reading this jitter results badly. I read 0.288 ms as 200ms, not 0.2ms - so it's not actually that bad at all. 🤦🏻 Please excuse my panic.

[XAMPPRocky]

Moving this to high priority because I'd to have to a criterion benchmark in the repository before release.

[XAMPPRocky]

Going to close this, as we now have a couple of benchmarks, and we can track any of them in newer more specific issues.