IoUring: Submit after completion queue was processed

[normanmaurer]

Motivation:

We should try to submit one more time after we did run all completions. Otherwise we might introduce latency that will hurt performance.

Modifications:

Add one more submit call.

Result:

Better troughput.

Before the change:

Destination: [127.0.0.1]:8088
Interface lo address [127.0.0.1]:0
Using interface lo to connect to [127.0.0.1]:8088
Ramped up to 1 connections.
Total data sent:     134066.2 MiB (140578586624 bytes)
Total data received: 134066.1 MiB (140578537472 bytes)
Bandwidth per channel: 74755.196⇅ Mbps (9344399.5 kBps)
Aggregate bandwidth: 37377.591↓, 37377.605↑ Mbps
Packet rate estimate: 3424327.9↓, 3208145.5↑ (12↓, 45↑ TCP MSS/op)
Test duration: 30.0883 s.

After this change:

Destination: [127.0.0.1]:8088
Interface lo address [127.0.0.1]:0
Using interface lo to connect to [127.0.0.1]:8088
Ramped up to 1 connections.
Total data sent:     250820.2 MiB (263004028928 bytes)
Total data received: 250812.7 MiB (262996131840 bytes)
Bandwidth per channel: 139956.971⇅ Mbps (17494621.4 kBps)
Aggregate bandwidth: 69977.435↓, 69979.536↑ Mbps
Packet rate estimate: 6406626.3↓, 6006391.8↑ (12↓, 45↑ TCP MSS/op)
Test duration: 30.0664 s.

[franz1981]

I like it, and makes sense

[franz1981]

The same could happen after we drain completion on writability changed

[normanmaurer]

The same could happen after we drain completion on writability changed

Yeah but these will be handled by this submit as well... we also don't want to call submit too often as it's a syscall

[franz1981]

Looks great 😃 you're rocking here - let's make this fly

[chrisvest]

How's it looking compared to NIO and epoll now, in this test?

[normanmaurer]

How's it looking compared to NIO and epoll now, in this test?

4-5% slower... almost there ;)