QUIC network protocol - will it fix packing queuing?

[Frooxius]

Hello everyone!

I have another test for you. This one is a bit long running and doesn't need a special build - but it does require you to enable certain options.

Build Version

Whatever latest public version of Resonite is - there's no separate branch

What is being tested

We've been long battling the throughput issues with the LNL protocol, resulting in packet queing and delays. One of the alternatives that might behave better is the QUIC protocol, which is natively supported in .NET 10.

We have made initial implementation of this protocol into Resonite so we can gauge how well it might perform and if it's worth investing more time into it.

Whether you're prone to queuing or not, it's good to gather some data on how it performs!

How to test

There are two parts to testing - hosts of headlesses need to explicitly configure QUIC for WAN connections and users need to enable QUIC as preferred protocol for connecting to them.

I'm user connecting to servers

By default, Resonite will not use QUIC to connect.

To enable this, toggle the option in Settings -> Network -> Prefer QUIC (experimental)

When connecting to a QUIC enabled server, it should now use this protocol. Always verify that it's actually used by checking the protocol in one of the user display items.

I'm hosting headless

QUIC will not work out of box for your headless (except on LAN). We didn't implement NAT punchthrough for it yet, so you need to configure a public IP manually for it to be available.

1. Make sure your system supports QUIC - there will be QUIC supported: True in the log

   • If QUIC is not supported, you might need to install dependencies. More here: https://learn.microsoft.com/en-us/dotnet/fundamentals/networking/quic/quic-overview#platform-dependencies

2. Configure public IP & open appropriate ports

   • Create/Modify Config.json file where your headless is installed and add this config
   • Make sure to replace the IP with the actual public IP of your server
   • Ensure appropriate ports are opened on your firewall

{
    "quicConfig" : {
        "publicIP" : "127.0.0.1"
    }
}

3. Specify port for QUIC
   • If you can only open specific ports rather than a whole range, specify the port to use for QUIC in the config for starting worlds
   • Each world you host will need a unique port - if you host multiple worlds, you need to specify unique port for each
   • Note: This option replaces the legacy forcePort setting

"forcePorts": {
    "lnl" : 12000,
    "quic": 12001,
    "tcp" : 12002
},

Once you have all these options configured, QUIC should be supported. If your session is public, you can easily verify this by going to:
https://go.resonite.com/session/

And checking of the "Alternative URLs" contains quic://<your-ip>:<port>

What to test

• Connect to sessions with QUIC
• Use Resonite as usual
• Do shenanigans and stuff that will tax the network
• Import assets and other files that will take time to transfer (can help test throughput)
• Switch to QUIC in sessions where you are queuing with LNL and see if it resolves

What to watch out for

• Connections failing or falling back to LNL even if everything is configured properly
• Random disconnects
• Packet queuing - world changes being delayed
  • IMPORTANT: QUIC currently doesn't track queuing through a metric in-game, you'll need to watch out for desync yourself (e.g. by users grabbing an item in the world and making other changes and seeing if they're delayed)
• World updates stopping completely
• Crashes & errors that don't seem to occur with LNL or Steam Networking Sockets

Known limitations

• .NET 10 doesn't support datagram extension for QUIC. This means voices & poses use reliable streams, albeit on a separate channel
  • Voices & poses can queue
  • Voices & poses can stutter and stop for a bit
• Implementation uses self-signed certificates
  • We have not implemented a trusted authority to ensure full security of the connection yet - we are waiting to see if it's worth investing more time into this - but first we want to see if the protocol itself performs well
  • Resonite still performs its own UserID authentication and will disconnect if this fails
  • While attack on this is unlikely (especially with small number of users likely using this), we want to be transparent about the current state of this protocol
  • If we transition to using this as main protocol, we'll implement proper certificate validation
• QUIC metrics are limited and don't include tracking of queuing - you'll need to observe them manually

Report your observations!

• For general performance of the protocol, post a message here so we can gather general feedback
• If you find bugs & exceptions, create a standard bug report on this GitHub

How long will testing go for?

There's no set end, this is an ongoing test.

The experimental protocol exists alongside other protocols, so there's no rush and it can be used as a fallback or default protocol if it works well for you.

If we get promising results, we'll likely invest more time into polishing QUIC and switching over to it as main implementation.

I like programming, can I help get adopted quicker?

Yes!

For us to fully lean into QUIC, we require the Datagram extension, which allows sending unreliable data like voice & poses. There is a library that adds this support, but it's heavily tied to .NET 7/8 and doesn't work correctly on .NET 10.

Official support for this seems to be planned for .NET 12, which is at the time of this post more than 1 year out.

If you'd like to build Datagram extension that works with .NET 10 (and upcoming .NET 11), this would help us adopt this sooner if we have promising results.

You can find more info in the issue here: #5458

[DanteTucker]

It took me a while to realize the quicConfig option goes in a separate Config.json file in the root of the headless and not the normal headless config files.

Unfortunately it seems none of the headless docker images are set up to create this file easily right now.

[TisFoolish]

It's really annoying that this is a separate config file in a separate location.

[Frooxius]

It's a global config, where the global settings (like port ranges) for protocols are defined.

It needs to be separate, because it's not related to any world configurations - those settings affect both graphical client & headless.

[TisFoolish]

It's more of that there are multiple config files (and that they have the same name) but are in separate locations. It'd be better if there was just one, canonical location. Personally, multiple files makes sense when its configs for different layers of a system.

The Linux desktop environments all used to do something similar (multiple locations for multiple files) and it was very, very annoying when trying to keep a backup of your settings..

[Frooxius]

@TisFoolish They serve different purposes and will be updated at different times and different systems, so unfortunately it can't be served by a single config file in a single location.

This is a bit off topic though for this. We use what config system we have, changing that is not in scope of QUIC testing, so I'd recommend making separate GitHub issue if you want to continue this (though heads up, if the request is to have a single config file, it'll probably just get closed, but we can look into other things / re-organization).

[ProbablePrime]

I do have an issue open for slightly improving this scenario. Its a little old and I don't have any energy to do it or discuss it right now, but I did want to cross-link.

#2752

Thanks to bredo for finding it

[RetrowaveHyena]

Currently QUIC networking on the client does not appear to work on Linux systems. From experimenting and testing, libmsquic is not available inside the steam pressure vessel, which prevents QUIC from operating.

This occurs even after installing libmsquic as a package (tested on Arch Linux), or even compiling libmsquic manually and installing manually.

[zahndy]

Managed to get it working on f44
libmsquic.so needed:
libnuma.so.1
libcrypto.so.3
libnl-route-3.so.200
libnl-3.so.200
libbpf.so.1
libxdp.so.1
those had to be copied and renamed, it gets the rest from /usr/lib/pressure-vessel/overrides/lib/

[art0007i]

@Hayden-Fluff try installing the debian 11 libs. Baplar posted links to them a little bit under my workaround comment. I'm hoping that those ones will work for everyone the same since it's under SLR

[Baplar]

I just tested it on my Debian 13 laptop.
When using the latest libmsquic.so from the Debian 13 package (2.5.9) I had issues with missing libs, the ones cited by zahndy.
However, when using the latest one from the Debian 11 package (2.4.8) I just needed libnuma.so.1 !
It seems that the other libs in that list are new dependencise that were introduced between these two versions.

Here are the two libs in question, freshly extracted from the Debian 11 packages that I mentioned in my previous comment.
msquic.tar.gz

[art0007i]

I've edited my workaround comment suggesting people to install the debian 11 libs. hopefully it will work for everyone (it should since it runs in slr3 right?)

[DanteTucker]

Following MS's instructions for building it off their github worked fine for me. No need to worry about distro specific things or sourcing it from unconventional means.
https://microsoft.github.io/msquic/msquicdocs/docs/BUILD.html

[mxjessie]

In case anyone else is getting bit by a mysterious inability of the server to advertise its QUIC-ness despite having everything else set up correctly, Config.json config parameters are case sensitive. oops lol

[art0007i]

Just tried a test where I upload a video and time how much time it takes for the host to fully receive it.

for a ~48MB video it took ~12.5 seconds over QUIC
and 1min 45sec for LNL

which is an order of magnitude quicker!

[RetrowaveHyena]

Dont have exact nums, but i observe much quicker uploads as well

[DanteTucker]

I hosted a session on my headless with about 12 users for a few hours, 4 of us connecting with quic.
There were no noticeable issues.

[bredo228]

File transfer over QUIC is a little fast

[noblereign]

My poor 50mbps internet gets absolutely pummeled by file transfers 😭😭😭 they were watching bee movie (2.7gb) and it was like i was lobotomized for 9 minutes
https://github.com/user-attachments/assets/b46a5cae-dc19-4a76-80aa-fac2446cf1eb

once the transfer finished though, everything went so smoothly i honestly forgot i was even using QUIC!!

[Frooxius]

This is likely the unfortunate side effect of not being able to use the datagram extension and assign priority to streams.

Priority is coming in .NET 11, so should be soon(ish), but datagram extension might be in .NET 12.

[Frooxius]

Can you try this again in the latest build? I backported the priority setting, I'm curious if this helps things with your connection.

[mxjessie]

I've been running a lively cacophony factory all day and we've been helping folks try out QUIC for testing. We tossed some quite large videos at each other for testing purposes, duplicate-joined the session a whole lot via LNL/QUIC/TCP, did some impolite things to the datamodel (one thousand debugvectors my beloved). It went well overall, with just a few hitches.

My setup is a dedicated headless host, i7-6700k, 24GB RAM, approximately 500 Mbit symmetric fiber in Seattle WA with good throughput and latency characteristics even at near-maximum upload/download speeds owing to a local bottleneck at my edge keeping throughput well below a point at which bufferbloat ever seems to arise. It lives on the same gigabit Ethernet network as my desktop/VR client. I'm not normally prone to queueing except in 30+ user sessions with high latency to the host and individual users within (EU party sessions seem to do it).

My rough initial observations thusfar:

• Transfers over my LAN are capable of saturating my gigabit Ethernet connection, which, AFAIK, LNL has never been capable of. I hadn't tried TCP at all until today during this testing.
• Transfers to other users seem to run significantly faster. Bulk uploads run at the speed of folks' network connections,.
• Intense congestion ("buffer bloat") seems to be a cause of extremely deleterious effects to a users' voice/motion streams if they're uploading/downloading near their internet connection speed limit, but this would warrant more investigation with folks that know their latency characteristics under heavy network load better/could maybe be tested with artificial network degradation schemes. This may not actually be a QUIC thing at all; it may be a Splittening-esque "it's faster now so we see issues because of that" thing.
• User interactions between QUIC and LNL seemed OK, no great disparities there were observed. This was in a casual hangout session, so not especially demanding in responsivity.
• As mentioned, ping, packet loss, queued messages, and possibly some other network stats, currently, do not work. Some of these may not make sense anymore anyhow.
• We observed a few odd world hangs/crashes tonight, and the logs are rife with QUIC exceptions, most of them two types noted below.
• On a few instances, users would join and their names would be invisible in my session list. I don't know if others experienced this ever, but I saw it with two different users, one on LNL, one on QUIC. Rejoining would resolve it.
• It is still possible to stuff up the asset queue if e.g. a large file is brought in and then abruptly deleted/interrupted in its transfer.
• Blocked-user behavior in a shared session seemed to function as normal.
• Config.json does not support FQDNs. It'd be nice if it did eventually. I can make a ticket about it.
• It was found that libnuma, a dependency for msquic on Linux, is GPL/LGPL licensed. It may be possible to build msquic without libnuma; if licensing is a concern here this may be worth exploring.

Network traffic overview of the last 7 days of regular session hosting on my headless (Monday and Wednesday events reach comparable peak user counts to tonight, albeit with less overt stress-testing):

Zoom in on just tonight's tomfoolery:

A 700MB video file uploaded at a time when the session was largely QUIC users, to test performance. This video transfers from my desktop to the headless at near-gigabit line rate, then completely fans out to everyone on QUIC in <2 minutes. It continued to upload to the few LNL users in the session for about 15 more minutes.

A rough timeline of events, relative to my local time:

• 13:45: Configs sorted, testing began
• 18:36: Headless world "soft crash", the headless automatically restarted the world without intervention
• 22:49: First headless world "hard hang", frozen and then killed & restarted by the external watchdog
• 22:49: Second headless world "hard hang", frozen and then killed & restarted by the external watchdog
• 22:50: Tried changing the engine Config.json to a FQDN, ran briefly, nobody could join over QUIC anymore (except me via LAN IP). Reverted, session wound down a bit for the night.

I can provide combined server and client logs for this session run but would prefer to do so out-of-band out of user discretion with this feature in its early state; let me know if you'd like copies of these. Some exceptions that stand out in frequency are:

6:34:49 PM.449	Exception receiving data from QUIC stream:
System.Net.Quic.QuicException: Connection aborted by peer (11).
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at FrooxEngine.QUIC_Connection.ReadData(QuicStream stream, Byte[] buffer, Int32 count) in /home/runner/work/FrooxEngine/FrooxEngine/FrooxEngine/FrooxEngine/Networking/Realtime Networking/QUIC/QUIC_Connection.cs:line 397
   at FrooxEngine.QUIC_Connection.ProcessIncomingData(QuicStream stream, CancellationToken cancellationToken) in /home/runner/work/FrooxEngine/FrooxEngine/FrooxEngine/FrooxEngine/Networking/Realtime Networking/QUIC/QUIC_Connection.cs:line 340

throughout sessions, and

6:34:49 PM.449	Exception receiving data from QUIC stream:
System.Net.Quic.QuicException: Connection aborted by peer (11).
   at System.Net.Quic.QuicStream.HandleEventShutdownComplete(_SHUTDOWN_COMPLETE_e__Struct& data)
   at System.Net.Quic.QuicStream.HandleStreamEvent(QUIC_STREAM_EVENT& streamEvent)
   at System.Net.Quic.QuicStream.NativeCallback(QUIC_HANDLE* stream, Void* context, QUIC_STREAM_EVENT* streamEvent)
--- End of stack trace from previous location ---
   at System.Net.Quic.ResettableValueTaskSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
   at System.Net.Quic.QuicStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at FrooxEngine.QUIC_Connection.ReadData(QuicStream stream, Byte[] buffer, Int32 count) in /home/runner/work/FrooxEngine/FrooxEngine/FrooxEngine/FrooxEngine/Networking/Realtime Networking/QUIC/QUIC_Connection.cs:line 397
   at FrooxEngine.QUIC_Connection.ProcessIncomingData(QuicStream stream, CancellationToken cancellationToken) in /home/runner/work/FrooxEngine/FrooxEngine/FrooxEngine/FrooxEngine/Networking/Realtime Networking/QUIC/QUIC_Connection.cs:line 340

whenever a user left the session.

I think some good next tests might be things like game or multimedia worlds that rely on/benefit from high speed and coordinated activities, along with more general testing with more marginal internet connections. This is exciting, so far, though!!

[Baplar]

According to the README of the numactl repo libnuma itself is under LGPL specifically, so it should be fine to redistribute.

[XDelta]

I've been using quic on windows 10 and seen no issues come up while using it so far. Aside from joining one testing specific session, it has just been my usual usage of the platform.

This did require swapping my dotnet runtime provided msquic.dll for a version built with openssl which can be obtained from https://www.nuget.org/packages/Microsoft.Native.Quic.MsQuic.OpenSSL. I added a tutorial for swapping them on the wiki for others who may want to experiment with that https://wiki.resonite.com/Guide_for_QUIC_on_Windows10.

It may also be a point of consideration towards using a version that would have wider support in the meantime although even as is, it isn't terribly difficult for a user to make it work less officially if they wish to.

[Frooxius]

Thanks for all the tests, investigations and help everyone!

So far from what I'm reading, QUIC seems very promising way forward, with greatly increased throughput over LNL.

I think this is a better way forward than trying to wrangle LNL to behave better and be able to push more data - even with large window size, it can't seem to match this protocol in its efficiency.

---

The main issue I see right now is that large transfers negatively affect voice and pose streams. There's a few solutions to this:

1. Ideal one - datagram extension - voices & poses don't need to use reliable streams.
   • However this one isn't planned until .NET 12
   • If anyone wants to look into replicating what the other library did to add this extension before then, this would help us a ton
2. Giving streams priority - already implemented in .NET 11. I could potentially see if we can sneak this in through Reflection / P/Invoke - I looked at the source and it might be doable
   • I'll try to sneak this into the next build if possible
   • This might work well enough in the interim

Other than that it's not working out of the box on graphical Linux client - but there's solutions for that we can integrate (I've asked @BlueCyro if he can integrate those).

[Frooxius]

Good news, I managed to backport the priority setting from .NET 11!

From the latest build release notes:

• Backported QUIC stream priority assignment from .NET 11 and set appropriate priorities
  -- Voice & Pose streams have highest priority
  -- World data model changes medium
  -- Asset transfers lowest
  -- This might help with some users experiencing degraded voice/poses when assets are transferring - please give it a test and see how well this works with this build!

[ohzee00]

Tested the above changes with a session that ran roughly 6 hours with 20 or so people in and out(with about half or slightly more using QUIC), local asset upload didn't saturate my 10mb connection and still allowed me to talk/receive packets from others.

Overall it worked damn well, I did notice hiccups in user audio every so often but it was brief and most likely due to the current lack of unreliable streams. No queuing behavior was seen with some testing (literally flashing context menus at eachother) along with very fast local asset transfers, with LNL connections taking whole minutes to download something compared to the QUIC connection types.

[MarkN2000]

I’ve had a QUIC-enabled session running since right after the release, and it’s been working fine so far.
https://api.resonite.com/open/session/S-U-1NzqeqewOpM:nokemono01

I’m hosting it on an ARM Linux instance on Oracle VPS; it worked properly once I installed the necessary modules as outlined in the Microsoft documentation.

One question I have is: does QUIC only support IPv4? With the configuration method I tried, it wouldn't start up correctly when I specified an IPv6 address.

[Frooxius]

QUIC should support IPv6 when fully implemented. With partial implementation right now for testing purposes it's done relatively quick to put the IP in, so that wasn't considered, but that can be fixed.