Investigate faf results on Plaintext

[sebastienros]

It's currently over 8M RPS which is above the theorical limit of the network.

[sebastienros]

I think the date is empty as the variable is never initialized: https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/faf/src/main.rs#L43

The initialization seems to have been removed in this PR: https://github.com/TechEmpower/FrameworkBenchmarks/pull/6523/files#diff-e828d6a53c980b3107862cfa2530ba397b76f56d2a0bb3d555465a6f6217f4feL52-L53

That would explain the network bandwidth difference.

[sebastienros]

/cc @errantmind

[sumeetchhetri]

Please look at the source for the framework here, the date value is passed along from the main event loop, also the date header validation does exist in the tfb toolset.

[sebastienros]

Thanks for confirming. Do you know how it's getting over the "physical" limits then? If not then I will try to check the full payload it's sending so I can understand.

[fafhrd91]

could process priority affect throughput?

[sebastienros]

@fafhrd91 historically all plaintext results have been capped at 7M rps because when you look at the minimum request size and the network packets, over a 10Gb/s network, it can't be faster than that.

This was confirmed by:

• stripping the /paintext url to /p (and triggered a new rule in the specs)
• using smaller Server: ... headers (and triggered a new rule in the specs)
• using a faster network card (in our Microsoft's lab) which improved all plaintext results for top frameworks (up to 14M rps) without changing anything else.

I am not excluding any magic trick though, even at the network level, which could be a totally fine explanation.

[billywhizz]

@sebastienros @fafhrd91 @sumeetchhetri @errantmind

i've done a little investigation here and it seems each faf request from wrk is 87 bytes and each response back is 126 bytes.

so, if we assume for argument sake an average ethernet packet size of 8192 bytes, that would allow 64 responses per packet plus 66 bytes of protocol overhead for TCP/IP/Ethernet, giving us an overhead of ~1 byte per response for an overall average including protocol overhead of 127 bytes per response.

if we divide 10,000,000,000 (10Gb) bits by (127 * 8 = 1016) bits we get a theoretical max throughput of 9,842,519 responses per second on a 10Gb ethernet interface.

so, i think it is possible that the faf solution, due to it's extreme low level and focus on shaving every possible microsecond off the server side processing overhead could be significantly faster than all the other frameworks no? it does seem to skew the results on the TE charts though so maybe it should be tagged as stripped instead of realistic?

i also can confirm that the responses are adhering to the rules and the date field does change at least every second.

[billywhizz]

@sebastienros are my calculations above correct? as far as i understand, they would assume full duplex communication between web server and benchmark server so that would mean theoretical throughput of 10Gb up and 10Gb down simultaneously yes? if not, then i can't understand how we could even reach 7m RPS. 🤷‍♂️

[errantmind]

I didn't see this right away, so sorry for the delay.

There is no single secret that accounts for the difference in performance between faf and other projects. Each component in faf has been hand optimized to some degree.

If anyone is extra curious they can test each component individually against what they have. The source code is a little dense but has decent separation-of-concerns.

[billywhizz]

@errantmind i will take a look. i find rust syntax so horrific to look at though. i'll see if i can put together something in plain old C that would achieve the same throughput.

@sebastienros @nbrady-techempower can anyone confirm re. my questions/calculations above? i had thought, as sebastien asserted above, that 7m RPS was some kind of hard limit based on the hardware but it seems not. 🤷‍♂️

[sebastienros]

Sorry, haven't had time to look into it, but I had done the same calculation 3 years ago (we recorded it I think) and we confirmed the 7M. I had looked at wireshark traces too because we didn't understand why all results capped at 7M. Then we ordered new NICs (40gb/s) and magically the benchmarks became much faster. All these benchmarks that are currently at 7M reach 10M+ when using a faster card, with the same hardware (we own the same physical machines as TE), so I don't think faf is just faster because it's "more" optimized than other fast frameworks. But until proven otherwise this is what it is, and I totally understand that there are some tests to validate the results. Hope to find some time to investigate more soon.

[sebastienros]

@billywhizz

if we assume for argument sake an average ethernet packet size of 8192 bytes

MTU is 1500 on these NICs, including the ones we use.

[billywhizz]

@sebastienros ok, so if my quick calculation is correct that would make it a theoretical max RPS of ~9.4m.

each packet = (11 * 126) + 66 = 1452 bytes = 11616 bits
10bn bits / 11616 = 860881 packets
860881 packets * 11 = 9,469,696 RPS

assuming full duplex and that downstream link can use the full 10Gb. does that sound correct?

[sebastienros]

I took some tcp dumps and all looks fine. Compared to ULib (c++) also which used to be the fastest on our faster network.

The important things I noted are

• The headers Server : F vs Server: ULib, which is saving 3 bytes.
• The packets are smaller for ULib, so it's more common to have the responses split in two packets vs. FAF
• All pipelined requests fit in one packet of 2720B, for both. Even though the MTU is 1500. I read there are some network optimizations that make it possible.

But something that you should re-calculate, is that the requests are 170B, so these should be the network bottleneck (if it is one):

From the client each packet is 2720B (16 requests of 170B) plus the preamble (66B) which makes 2786B to send 16 requests.
The NICs are full-duplex so 10Gb up AND down.
10^9/2786 = 358937 groups of 16 requests per second
358937 * 16 = 5,742,992 RPS

And this is assuming there is no lost/empty packets (and there are)

[billywhizz]

@sebastienros

thanks for getting back. that's odd - when i run locally and do a single http request the http payload is only 126 bytes. not sure where 170 bytes is coming from. thanks for confirming re. full duplex. i think my calculation is correct then if we can resolve the 126 bytes versus 170 bytes you are seeing for http payload?