feat: Faster UDP/IO on Apple platforms by larseggert · Pull Request #1993 · quinn-rs/quinn

larseggert · 2024-09-20T10:25:25Z

This uses Apple's private sendmsg_x and recvmsg_x system calls for multi-packet UDP I/O.

Ralith · 2024-09-20T19:40:32Z

Is there interest in seeing TX support via sendmsg_x?

We found there wasn't much performance benefit, and was considerable difficulty taking advantage of, sendmmsg-style batching. IIRC the _x functions on macOS have more to offer than that, though. Will this unblock segmentation offload or other incidental optimizations?

larseggert · 2024-09-23T06:02:47Z

Bench on main:

test large_data_10_streams  ... bench:  27,558,791 ns/iter (+/- 13,459,810) = 380 MB/s
test large_data_1_stream    ... bench:  24,324,266 ns/iter (+/- 19,219,937) = 43 MB/s
test small_data_100_streams ... bench:  19,437,900 ns/iter (+/- 20,065,941)
test small_data_1_stream    ... bench:  11,465,128 ns/iter (+/- 8,699,934)

Bench with this PR:

test large_data_10_streams  ... bench:  28,829,216 ns/iter (+/- 15,924,956) = 363 MB/s
test large_data_1_stream    ... bench:  14,354,999 ns/iter (+/- 20,039,122) = 73 MB/s
test small_data_100_streams ... bench:  14,061,741 ns/iter (+/- 17,311,517)
test small_data_1_stream    ... bench:  19,194,441 ns/iter (+/- 5,012,070)

Surprised that large_data_10_streams and small_data_1_stream are slower...

Ralith · 2024-09-23T06:34:07Z

Those tests tend to be extremely noisy, as the huge variance suggests. A targeted quinn-udp benchmark might be more useful.

larseggert · 2024-09-23T06:42:37Z

We've also found on neqo that multi-packet RX without multi-packet TX has limited benefits, since the RX batch size will be very small.

larseggert · 2024-09-23T07:29:18Z

I added sendmsg_x support, mostly to see what the performance difference would be. But it seems that none of the benches or tests call send with a Transmit struct where segment_size is not None?

mxinden · 2024-09-23T09:36:20Z

A targeted quinn-udp benchmark might be more useful.

How about using the throughput.rs benchmark @larseggert?

https://github.com/quinn-rs/quinn/blob/main/quinn-udp/benches/throughput.rs

larseggert · 2024-09-23T12:32:18Z

With @mxinden's benchmark. Baseline:

gso_true/throughput     time:   [58.076 ms 58.230 ms 58.387 ms]
                        thrpt:  [171.27 MiB/s 171.73 MiB/s 172.19 MiB/s]

Only sendmsg_x:

gso_true/throughput     time:   [15.143 ms 15.189 ms 15.236 ms]
                        thrpt:  [656.35 MiB/s 658.36 MiB/s 660.37 MiB/s]
                 change:
                        time:   [-74.028% -73.915% -73.808%] (p = 0.00 < 0.05)
                        thrpt:  [+281.80% +283.36% +285.04%]
                        Performance has improved.

Both sendmsg_x and recvmsg_x:

gso_true/throughput     time:   [12.632 ms 12.682 ms 12.731 ms]
                        thrpt:  [785.46 MiB/s 788.53 MiB/s 791.61 MiB/s]
                 change:
                        time:   [-78.321% -78.221% -78.112%] (p = 0.00 < 0.05)
                        thrpt:  [+356.88% +359.16% +361.27%]
                        Performance has improved.

Both sendmsg_x and recvmsg_x with BATCH_SIZE of 64:

gso_true/throughput     time:   [11.640 ms 11.682 ms 11.725 ms]
                        thrpt:  [852.85 MiB/s 856.00 MiB/s 859.07 MiB/s]
                 change:
                        time:   [-80.030% -79.938% -79.844%] (p = 0.00 < 0.05)
                        thrpt:  [+396.13% +398.45% +400.75%]
                        Performance has improved.

djc

LGTM.

Please squash all of the changes into a single commit?

mxinden

Impressive results. Great to see the MacOS _x syscalls work for QUIC UDP IO.

Ralith

Can we enable real GSO/GRO using these interfaces?

larseggert · 2024-09-24T04:51:53Z

No. They are the equivalent of the mmsg Linux calls. AFAIK Apple doesn't have GSO/GRO via the socket interface.

larseggert · 2024-10-01T16:24:34Z

Are you waiting on anything from me on this?

mxinden

quinn-udp/benches/throughput.rs will need more changes to still support non-apple platform. @larseggert I believe we will need to either run it multi-threaded, or use some kind of executor, e.g. tokio. I can prepare a commit in the next couple of days. Sorry for missing this in earlier reviews.

Changes itself look good to me.

Done.

djc · 2024-10-09T14:32:56Z

@Ralith can you do another round on this one?

larseggert · 2024-10-10T05:16:33Z

Once @mxinden's fix to the bench is in, I will rebase and squash this PR.

AndrewDryga · 2024-10-10T17:43:56Z

Hey guys 👋, is there any chance Apple won't approve apps that are using those private syscalls in the App Store? They are notorious for doing so and even de-listing apps for using anything "undocumented". See 2.5.1 here: https://developer.apple.com/app-store/review/guidelines/

See one of such cases here: https://9to5mac.com/2019/11/04/electron-app-rejections/

How they will find out? Apple employs automated tools to scan apps for the usage of private APIs. If sendmsg_x and recvmsg_x are detected, the app is at risk of being flagged.

larseggert · 2024-10-10T17:49:20Z

The use of the private syscalls is now behind a non-default feature.

AndrewDryga · 2024-10-10T17:51:22Z

@larseggert should we add a big fat warning saying that if you enable this flag you will violate Apple ToS so it's only should be enabled if app is not distributed via App Store (or notarized for EU)?

larseggert · 2024-10-24T12:16:23Z

Shouldn't the features CI workflow have included fast-apple-datapath? I don't see it in the logs. (Or I am missing something.)

mxinden · 2024-10-24T12:33:52Z

Should be fine I believe:

running `cargo check --no-default-features --features direct-log,fast-apple-datapath,log,tracing` on quinn-udp (1754/1772)
running `cargo check --no-default-features --features default` on quinn-udp (1755/1772)
running `cargo check --no-default-features --features direct-log` on quinn-udp (1756/1772)
running `cargo check --no-default-features --features default,direct-log` on quinn-udp (1757/1772)
running `cargo check --no-default-features --features fast-apple-datapath` on quinn-udp (1758/1772)
running `cargo check --no-default-features --features default,fast-apple-datapath` on quinn-udp (1759/1772)
running `cargo check --no-default-features --features direct-log,fast-apple-datapath` on quinn-udp (1760/1772)
running `cargo check --no-default-features --features default,direct-log,fast-apple-datapath` on quinn-udp (1761/1772)
running `cargo check --no-default-features --features log` on quinn-udp (1762/1772)
running `cargo check --no-default-features --features direct-log,log` on quinn-udp (1763/1772)
running `cargo check --no-default-features --features fast-apple-datapath,log` on quinn-udp (1764/1772)
running `cargo check --no-default-features --features direct-log,fast-apple-datapath,log` on quinn-udp (1765/1772)
running `cargo check --no-default-features --features tracing` on quinn-udp (1766/1772)
running `cargo check --no-default-features --features direct-log,tracing` on quinn-udp (1767/1772)
running `cargo check --no-default-features --features fast-apple-datapath,tracing` on quinn-udp (1768/1772)
running `cargo check --no-default-features --features direct-log,fast-apple-datapath,tracing` on quinn-udp (1769/1772)
running `cargo check --no-default-features --features log,tracing` on quinn-udp (1770/1772)
running `cargo check --no-default-features --features direct-log,log,tracing` on quinn-udp (1771/1772)
running `cargo check --no-default-features --features fast-apple-datapath,log,tracing` on quinn-udp (1772/1772)

Currently we use `quinn-udp` `v0.5.4`. `quinn-udp` `v0.5.5` fixes [`recvmmsg` calls on Android x86](quinn-rs/quinn#1966). `quinn-udp` `v0.5.6` adds [experimental multi-message support on Apple platforms](quinn-rs/quinn#1993) and [fixes an unnecessary `windows-sys` version restriction](quinn-rs/quinn#2021). While not strictly necessary, given that our current version specification (i.e. `version = "0.5.4"`) already allows users to use Neqo with `quinn-udp` `v0.5.6`, this commit updates to `quinn-udp` `v0.5.6` anyways, thus making sure CI tests with latest version. In case mozilla#2208 lands, future compatible version updates would touch the `Cargo.lock` file, not `Cargo.toml`.

Currently we use `quinn-udp` `v0.5.4`. `quinn-udp` `v0.5.5` fixes [`recvmmsg` calls on Android x86](quinn-rs/quinn#1966). `quinn-udp` `v0.5.6` adds [experimental multi-message support on Apple platforms](quinn-rs/quinn#1993) and [fixes an unnecessary `windows-sys` version restriction](quinn-rs/quinn#2021). While not strictly necessary, given that our current version specification (i.e. `version = "0.5.4"`) already allows users to use Neqo with `quinn-udp` `v0.5.6`, this commit updates to `quinn-udp` `v0.5.6` anyways, thus making sure CI tests with latest version. In case #2208 lands, future compatible version updates would touch the `Cargo.lock` file, not `Cargo.toml`.

Currently we use `quinn-udp` `v0.5.4`. `quinn-udp` `v0.5.5` fixes [`recvmmsg` calls on Android x86](quinn-rs/quinn#1966). `quinn-udp` `v0.5.6` adds [experimental multi-message support on Apple platforms](quinn-rs/quinn#1993) and [fixes an unnecessary `windows-sys` version restriction](quinn-rs/quinn#2021). While not strictly necessary, given that our current version specification (i.e. `version = "0.5.4"`) already allows users to use Neqo with `quinn-udp` `v0.5.6`, this commit updates to `quinn-udp` `v0.5.6` anyways, thus making sure CI tests with latest version. In case #2208 lands, future compatible version updates would touch the `Cargo.lock` file, not `Cargo.toml`. Co-authored-by: Lars Eggert <lars@eggert.org>

mxinden · 2024-11-27T19:26:14Z

+        io::ErrorKind::Interrupted => {
+            // Retry the transmission
+        }


Given that this is not executed in a loop, it doesn't actually retry, but instead drop the packet to be sent.

We do retry on Linux:

quinn/quinn-udp/src/unix.rs

Lines 311 to 314 in 2edf192

io::ErrorKind::Interrupted => {

// Retry the transmission

continue;

}

But not on the BSDs or the slow Apple:

quinn/quinn-udp/src/unix.rs

Lines 433 to 435 in 2edf192

io::ErrorKind::Interrupted => {

// Retry the transmission

}

I don't recall whether we discussed this. @larseggert was this by design?

I think this was an oversight. It would be good to have identical behavior.

Agreed. @larseggert I will follow-up with a pull request.

With quinn-rs#2017, the concrete `send` implementations per platform are supposed to propagate `io::Error`s. Those errors are then eiter logged and dropped in `UdpSocketState::send` or further propagated in `UdpSocketState::try_send`. The `fast_apple` `send` implementation added in quinn-rs#1993 does not follow this pattern. This commit adjusts the `fast_apple` implementation accordingly.

With #2017, the concrete `send` implementations per platform are supposed to propagate `io::Error`s. Those errors are then eiter logged and dropped in `UdpSocketState::send` or further propagated in `UdpSocketState::try_send`. The `fast_apple` `send` implementation added in #1993 does not follow this pattern. This commit adjusts the `fast_apple` implementation accordingly.

mxinden · 2025-03-09T15:09:49Z

For those interested in using the fast-apple-datapath feature:

All MacOS Firefox Nightly users are now using it.
We see ~11% performance improvement on our localhost CPU bound throughput benchmark.
Every other (50th percentile) QUIC UDP recvmsg_x call reads 2 datagrams.
We are soon rolling it out to all Firefox Beta users. Once that succeeds it will make it into the Firefox release channel.

https://glam.telemetry.mozilla.org/fog/probe/networking_http_3_udp_datagram_segments_received/explore?aggType=avg&os=Darwin&timeHorizon=WEEK&visiblePercentiles=%5B99.9%2C99%2C95%2C75%2C50%2C25%2C5%5D

Ralith reviewed Sep 20, 2024

View reviewed changes

Comment thread quinn-udp/build.rs Outdated

larseggert marked this pull request as ready for review September 23, 2024 09:10

djc approved these changes Sep 23, 2024

View reviewed changes

Comment thread quinn-udp/benches/throughput.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

mxinden reviewed Sep 23, 2024

View reviewed changes

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Ralith previously requested changes Sep 23, 2024

View reviewed changes

Comment thread quinn-udp/benches/throughput.rs Outdated

djc reviewed Sep 24, 2024

View reviewed changes

Comment thread quinn-udp/src/cmsg/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

mxinden reviewed Sep 24, 2024

View reviewed changes

Comment thread quinn-udp/src/unix.rs Outdated

Ralith reviewed Sep 25, 2024

View reviewed changes

Comment thread quinn-udp/benches/throughput.rs Outdated

Comment thread quinn-udp/benches/throughput.rs Outdated

larseggert requested a review from Ralith October 1, 2024 16:24

larseggert mentioned this pull request Oct 3, 2024

perf: don't allocate in UDP send & recv path mozilla/neqo#2093

Closed

mxinden reviewed Oct 3, 2024

View reviewed changes

Comment thread quinn-udp/benches/throughput.rs Outdated

Comment thread quinn-udp/benches/throughput.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs

mxinden mentioned this pull request Oct 8, 2024

GSO on macOS #1554

Open

mxinden reviewed Oct 8, 2024

View reviewed changes

Comment thread quinn-udp/src/unix.rs Outdated

Comment thread quinn-udp/src/unix.rs Outdated

mxinden mentioned this pull request Oct 8, 2024

bench(udp): support all GSO and recvmmsg permutations across platforms larseggert/quinn#1

Closed

mxinden mentioned this pull request Oct 10, 2024

bench(udp): run GSO, GRO and recvmmsg permutations #2010

Merged

djc added this pull request to the merge queue Oct 25, 2024

Merged via the queue into quinn-rs:main with commit adc4a06 Oct 25, 2024

larseggert deleted the feat-apple-datapath branch October 25, 2024 07:29

mxinden mentioned this pull request Oct 28, 2024

chore(quinn-udp): increase crate patch version to v0.5.6 #2022

Merged

mxinden mentioned this pull request Oct 30, 2024

chore: update to quinn-udp v0.5.6 mozilla/neqo#2209

Merged

thomaseizinger mentioned this pull request Nov 1, 2024

feat(connlib): utilise GSO for UDP sockets firezone/firezone#7210

Merged

psumbera added a commit to psumbera/quinn that referenced this pull request Nov 5, 2024

Unbreak Solaris build after quinn-rs#1993

9ef3057

psumbera added a commit to psumbera/quinn that referenced this pull request Nov 5, 2024

Unbreak Solaris build after quinn-rs#1993

79453bc

djc pushed a commit that referenced this pull request Nov 5, 2024

Unbreak Solaris build after #1993

722835b

djc mentioned this pull request Nov 11, 2024

Unable to build quinn-udp (0.5.7) for NetBSD: error[E0428]: the name "recv" is defined multiple times #2033

Closed

mxinden mentioned this pull request Nov 16, 2024

perf: use recvmmsg in addition to GRO mozilla/neqo#2137

Closed

mxinden reviewed Nov 27, 2024

View reviewed changes

mxinden mentioned this pull request Nov 29, 2024

fix(udp): propagate error on apple_fast #2073

Merged

mxinden mentioned this pull request Dec 10, 2024

perf(udp): enable multi-message IO on apple mozilla/neqo#2279

Closed

mxinden mentioned this pull request Dec 23, 2024

perf(udp): multi-message receive on apple mozilla/neqo#2302

Merged

mxinden mentioned this pull request Jan 9, 2025

ci: run quinn-udp tests with fast-apple-datapath #2130

Merged

mxinden mentioned this pull request Apr 3, 2025

Relax MAX_TRANSMIT_SEGMENTS #2189

Open

	io::ErrorKind::Interrupted => {
	// Retry the transmission
	continue;
	}

Uh oh!

Conversation

larseggert commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ralith commented Sep 20, 2024

Uh oh!

Uh oh!

larseggert commented Sep 23, 2024

Uh oh!

Ralith commented Sep 23, 2024

Uh oh!

larseggert commented Sep 23, 2024

Uh oh!

larseggert commented Sep 23, 2024

Uh oh!

mxinden commented Sep 23, 2024

Uh oh!

larseggert commented Sep 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

djc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ralith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

larseggert commented Sep 24, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

larseggert commented Oct 1, 2024

Uh oh!

mxinden left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

djc commented Oct 9, 2024

Uh oh!

larseggert commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AndrewDryga commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larseggert commented Oct 10, 2024

Uh oh!

AndrewDryga commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larseggert commented Oct 24, 2024

Uh oh!

mxinden commented Oct 24, 2024

Uh oh!

mxinden Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

larseggert commented Sep 20, 2024 •

edited

Loading

larseggert commented Sep 23, 2024 •

edited

Loading

larseggert commented Oct 10, 2024 •

edited

Loading

AndrewDryga commented Oct 10, 2024 •

edited

Loading

AndrewDryga commented Oct 10, 2024 •

edited

Loading

mxinden commented Mar 9, 2025 •

edited

Loading