Latency and jitter only in moonlight, not observed during bandwidth test

[maretodoric]

READ ME FIRST!
If you're here because something basic is not working (like gamepad input, video, or similar), it's probably something specific to your setup, so make sure you've gone through the Troubleshooting Guide first: https://github.com/moonlight-stream/moonlight-docs/wiki/Troubleshooting

If you still have trouble with basic functionality after following the guide, join our Discord server where there are many other volunteers who can help (or direct you back here if it looks like a Moonlight bug after all). https://moonlight-stream.org/discord

Describe the bug
I'm facing some strange issue trying to stream from my PC to Raspberry pi 4 4GB.

• RPi is connected on 5GHz WiFi, router is fairly close (signal strength 85% as per LibreELEC, -50dbm reported by OpenWRT)
• OS installed is LibreELEC with moonlight add-on.
• Moonlight-qt is set to use 50000 bitrate (tried all ranges from 10000-100000), 780P
  When i launch a game from RPi, occasionally - every few seconds video and audio will stutter. And it's on repeat.

I cannot figure out where the issue is but i sincerely doubt the network since I'm known to watch 4K videos from Netflix without any issues and when running tests using iperf between pi and PC i get steady and stable results.

Steps to reproduce
Run the game

Screenshots
None, problem not visible on a screenshot

Affected games
Mortal Kombat 11, Steam Big Picture

Other Moonlight clients
Haven't tested

Moonlight settings (please complete the following information)

• Have any settings been adjusted from defaults?
  Launch parameters adjusted to --quit-after --720 --no-audio-on-host --bitrate 50000
• If so, which settings have been changed?
  bitrate was adjusted between 10000 - 100000 without any performance impact
• Does the problem still occur after reverting settings back to default?
  Yes

Gamepad-related issues (please complete if problem is gamepad-related)

Client PC details (please complete the following information)

• OS: LibreELEC 10
• Moonlight Version: 3.2.0
• GPU: Raspberry Pi 4 - Not sure which GPU is it using

Server PC details (please complete the following information)

• OS: Windows 10 21H2
• GeForce Experience version: 3.24.0.126
• Nvidia GPU driver: 497.29
• Antivirus and firewall software: Windows Firewall, Kaspersky Internet Security (turned off)

Moonlight Logs (please attach)
https://pastebin.com/vNQJNyNA

Additional context
I performed following testing using iperf program between client and server

• This is test ran when i was not streaming anything using moonlight, so idle: https://pastebin.com/23VnNpJP
  We can see 0 jitter for sender and 0.202ms jitter for receiver both 0 packet loss, consistent speed of 50000K bitrate
• This is test i ran while i was streaming, so test was running in the background while game was running in foreground in moonlight: https://pastebin.com/pVAHiYx6
  Here we can see inconsistent speed probably due to one stream already running, however jitter is again 0.000 ms for sender and 0.061 ms jitter for receiver however this time we can see 0.056% packet loss which is miserable if you would agree considering UDP datagram.
• Now this is where it gets really weird, moonlight log shows a lot higher amount of dropped packets and jitter
  Frames dropped by your network connection: 16.44%
  Frames dropped due to network jitter: 14.11%

I would really appreciate assistance to get to the bottom of this.

[cgutman]

These are some of the hardest issues to diagnose, so it may not be easy/possible to understand exactly what's going on.

However, my guess is that either your Pi or a switch between your host PC and your Pi doesn't cope well with GameStream's traffic pattern. Unlike iperf which is sending a uniform 50 Mbps distributed equally over time, the host PC using GameStream is sending batches packets as fast as it can every 16ms, then nothing, then another batch of packets, etc. The instantaneous bandwidth during those bursts can be far more than the selected "average" bitrate. Some hardware doesn't have the buffering capacity to deal with that kind of load without dropping packets.

however this time we can see 0.056% packet loss which is miserable if you would agree considering UDP datagram.

GameStream has FEC (error correction) data included in the stream, which should be more than enough to compensate for such a very small loss rate. The key again is those bursts of packets though. For example, if you lose 1 packet from every frame, Moonlight will recover 100% of frames successfully using that extra FEC data. However, if you lose 100% of packets from every 1/100 frames (just ~1% packet loss overall), then you'll have a 1% frame loss rate, even though this scenario has a much lower average packet loss rate than the prior one. Talking about averages can be misleading in this way.

You can try a different switch or try lowering your PC's NIC speed to 100 Mbps Full Duplex. If either of those solve it, then we have the right cause.

[maretodoric]

Thanks for taking time to reply! This helps me narrow it down really.
My Pi is connected to Xiaomi Mi Gigabit Router via 5GHz WiFi. Which is connected to main mikrotik to which another mikrotik is connected where my PC is connected on 1Gbps link.. So packets are jumping a lot 🤦
I cannot connect directly to MT where PC connected since it's far away for 5GHz and even far away for a cable...

But I'll try something else:

• As you suggested, I'll lower the PC link to 100mbps full duplex
• If that doesn't work, I'll see if mi router can catch 5Ghz network from MT that is connected to my pc. If it can I'll connect to it that way and try to form a bridgde between pi and MT.. Not sure how will that work or will it work at all but might try.

Thanks again for clarifying, i hate troubleshooting these performance issues

[pogojotz]

@maretodoric did you find out something interesting? I might have similar issues with my FritzBox router.

[maretodoric]

I purchased mikrotik cap ac to replace xiaomi router. Slight performance improvement but issue is still there. I'm suspecting one too many hops to PC. Haven't yet had the chance to test the direct cable connection. It's not yet properly wired in that room.

[gschintgen]

Hi, I'm in the process of troubleshooting similar issues. Since I'm not the original reporter and the issue may not have the same cause, I'll try to keep it as constructive as I can by showing the actual commands I used.

Host: rather modern PC (13th gen Intel, AMD RX 6650XT, Ubuntu 22.04.2, mesa-kisak, latest Sunshine)
Client A: Thinkpad. No issues as far as I can tell, needs more playing though in order to confirm with absolute certainty.
Client B1: Raspi 4 with retropie on Buster and moonlight-embedded.
Client B2: same(!) Raspi 4 but with Raspberry Pi OS Bullseye, no desktop, only moonlight-qt.
Client C: SteamLink device (integrated NIC is 100Mbps only)
Network: everything Cat. 7 gigabit with Netgear Smart Switch GS724Tv4 and working perfectly according to "regular" iperf3 (see below). The switch should also be rather beefy for home usage...

On B1 (but not B2) and on C I seemingly keep getting lost frames which result in a partially garbled screen which take some time (i.e. screen content refresh) to resolve. On my Steam Link this happens ca. 2-4 times per hour. Otherwise it's perfect.

iperf3 -c desktop -u -R -b xxx works fine for high multiple hundred Megabits without issue. (TCP reaches theoretical GBit speeds anyway.)

Where it gets interesting is when I try to produce a "brutal" microbursty stream instead of iperf3's rather well-behaved default, the theory being, that the bandwidth should be sent in bursts at a 60Hz frequence, i.e. with interval of 17ms. Here's some interesting output that I could provoke on client B2 (which works fine for 20Mbps moonlighting AFAICT, but it's all in a constant state of flux and I don't have that much time to always reconfirm everything since it can take up to 20 minutes or so to show problems.)

pi@moonlight:~ $ iperf3 -c desktop -O 1 -u -R -b 80M -l 1000 --pacing-timer 17000 -t0                                                                [7/192]Connecting to host desktop, port 5201
Reverse mode, remote host desktop is sending
[  5] local 192.168.42.66 port 43493 connected to 192.168.42.29 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  9.46 MBytes  79.4 Mbits/sec  0.004 ms  465/10388 (4.5%)  (omitted)
[  5]   0.00-1.00   sec  9.39 MBytes  78.8 Mbits/sec  0.001 ms  190/10040 (1.9%)
[  5]   1.00-2.00   sec  9.27 MBytes  77.8 Mbits/sec  0.031 ms  141/9860 (1.4%)
[  5]   2.00-3.00   sec  4.19 MBytes  35.1 Mbits/sec  0.006 ms  5527/9917 (56%)
[  5]   3.00-4.00   sec  6.73 MBytes  56.5 Mbits/sec  0.006 ms  3071/10132 (30%)
[  5]   4.00-5.00   sec  9.35 MBytes  78.5 Mbits/sec  0.002 ms  225/10034 (2.2%)
[  5]   5.00-6.00   sec  9.39 MBytes  78.8 Mbits/sec  0.006 ms  178/10027 (1.8%)
[  5]   6.00-7.00   sec  9.48 MBytes  79.5 Mbits/sec  0.002 ms  82/10018 (0.82%)
[  5]   7.00-8.00   sec  9.34 MBytes  78.3 Mbits/sec  0.008 ms  76/9865 (0.77%)
[  5]   8.00-9.00   sec  9.45 MBytes  79.3 Mbits/sec  0.006 ms  122/10036 (1.2%)
[  5]   9.00-10.00  sec  7.75 MBytes  65.0 Mbits/sec  0.005 ms  1797/9926 (18%)
[  5]  10.00-11.00  sec  4.06 MBytes  34.0 Mbits/sec  0.005 ms  5774/10026 (58%)
[  5]  11.00-12.00  sec  4.07 MBytes  34.2 Mbits/sec  0.022 ms  5867/10136 (58%)
[  5]  12.00-13.00  sec  4.01 MBytes  33.6 Mbits/sec  0.090 ms  5654/9861 (57%)
[  5]  13.00-14.00  sec  3.98 MBytes  33.4 Mbits/sec  0.006 ms  5754/9929 (58%)
[  5]  14.00-15.00  sec  4.32 MBytes  36.3 Mbits/sec  0.004 ms  5598/10133 (55%)
[  5]  15.00-16.00  sec  9.35 MBytes  78.4 Mbits/sec  0.005 ms  229/10029 (2.3%)
[  5]  16.00-17.00  sec  9.36 MBytes  78.5 Mbits/sec  0.005 ms  216/10031 (2.2%)
[  5]  17.00-18.00  sec  9.49 MBytes  79.6 Mbits/sec  0.001 ms  86/10036 (0.86%)
[  5]  18.00-19.00  sec  9.24 MBytes  77.5 Mbits/sec  0.047 ms  171/9857 (1.7%)
[  5]  19.00-20.00  sec  9.59 MBytes  80.5 Mbits/sec  0.007 ms  42/10103 (0.42%)
[  5]  20.00-21.00  sec  9.51 MBytes  79.8 Mbits/sec  0.002 ms  0/9977 (0%)
[  5]  21.00-22.00  sec  9.49 MBytes  79.6 Mbits/sec  0.006 ms  0/9956 (0%)
[  5]  22.00-23.00  sec  9.53 MBytes  80.0 Mbits/sec  0.003 ms  0/9997 (0%)
[  5]  23.00-24.00  sec  9.53 MBytes  80.0 Mbits/sec  0.003 ms  0/9994 (0%)
[  5]  24.00-25.00  sec  9.58 MBytes  80.4 Mbits/sec  0.001 ms  0/10048 (0%)
[  5]  25.00-26.00  sec  9.57 MBytes  80.3 Mbits/sec  0.002 ms  0/10038 (0%)
[  5]  26.00-27.00  sec  9.53 MBytes  80.0 Mbits/sec  0.002 ms  0/9995 (0%)
[  5]  27.00-28.00  sec  9.51 MBytes  79.8 Mbits/sec  0.086 ms  0/9973 (0%)
[  5]  28.00-29.00  sec  9.52 MBytes  79.9 Mbits/sec  0.002 ms  0/9987 (0%)
[  5]  29.00-30.00  sec  9.55 MBytes  80.1 Mbits/sec  0.127 ms  0/10016 (0%)
[  5]  30.00-31.00  sec  9.55 MBytes  80.2 Mbits/sec  0.002 ms  0/10019 (0%)
[  5]  31.00-32.00  sec  9.47 MBytes  79.4 Mbits/sec  0.093 ms  0/9928 (0%)

Note the all important --pacing-timer option.

Anyone guessing what happened around the 20 second mark?

On the host I enabled traffic shaping using Linux's tc facility! I thought I finally had a solution to my woes, in particular for my Steam Link/moonlight-qt which also has similar issues. This could be quite logical: The host sends its traffic at gigabit speeds in aggressive microbursts to the switch which then has to buffer in order to transmit to the 10 times slower Steam Link NIC. And it's not the only traffic. (Maybe my NAS is doing some updates or sending some offsite backup, all potentially impacting buffers too. What do I know...)

Unfortunately I still had those occasional garbled images on my Steam Link when I finally tried this combination for actual moonlight gameplay. Bummer.

Next I'll (again) try reducing my host's speed to 100Mbps using ethtool -s enp6s0 speed 100. If I remember correctly this had beneficial effects on iperf3 to my Raspi. But experimenting with a slowed down interface on the Raspi itself led to strange behaviour (where it lost connection afterwards and such and originally it was connected to a Fritzbox and then only to the main Netgear switch, lots of trial and error.)

Back to being constructive. Here's my experimental traffic shaping script responsible for the improved iperf output above (edit: rate/burst may have had different values, can't remember):

#/bin/sh
set -ex
# traffic control (shaping) for Sunshine udp video stream (needs sudo)
IF=enp6s0
#RATE=100Mbit
RATE=50Mbit
BURST=64kbit
SPORT=5201 #(for iperf3)
#SPORT=47998 (for sunshine video)
tc qdisc del dev $IF root
tc qdisc add dev $IF root handle 1:0 htb default 1
tc class add dev $IF parent 1:0 classid 1:3 htb rate $RATE burst $BURST
tc filter add dev $IF parent 1:0 protocol ip prio 1 u32 match ip sport $SPORT 0xffff flowid 1:3

Quite simple, after RTFMing for a couple of hours ;-)

On a further note, RPi4 seems to have problems with UDP traffic as noted here. Maybe this explains the performance difference between clients B1 and B2 (newer kernel and OS) above.

What is frustrating is that the two clients that are most important to me are the problematic B1 and C. What's more, neither moonlight-embedded on B1 nor moonlight-qt on C have support for OSD statistics. Else I could modify my traffic shaping commands on-the-fly and watch for packet loss differences right in moonlight. So that's definitely my top priority feature request, if technically possible of course. Both apps can print "slow connection" messages though, so some debugging output should be possible even if it comes at a performance penalty. Maybe print some stats once every second? It'd be so valuable!

I hope the above gives at least some pointers to other users on how to diagnose issues with bursty UDP traffic, even though I could not yet solve my issues.
(One step that I'll also try is mirroring my host's port on the switch to another port and sniff it using wireshark to check if traffic shaping is actually doing what it's intended to do. Potentially the buffer overrun, hence lost packets/frames is happening on the SteamLink device itself. Then there's not much I could do. Or can I increase its receiving network buffers via ssh? One more thing to check...)

(I'm not 100% certain though that networking is even the real culprit. IIRC I had better results on B1 when I used software encoding instead of VAAPI. It's a pain to debug this. So many variables.)

Unfortunately the only alternative switch that I could try is another semi-managed Netgear switch (with potentially similar behavior all around) or maybe wire up my network so that host & client exclusively use my FritzBox 6591's switch instead of my main Netgear switch.

[gschintgen]

Update: I've just realized due to other issues on the tracker, that dropped frames are supposed to show up in the logs but I had nothing relevant in the -verbose/-debug logs on my client "B1", i.e. Pi4 on Buster/retropie/moonlight-embedded. What is perplexing is that apparently the common denominator of my two problematic clients (B1: Pi4 buster/moonlight-embedded, C: SteamLink/moonlight-qt) is that they only support h264! It could well be an encoder bug on AMD/VAAPI side which only happens with h264. In that case I'd be out of luck for Steam Link, except (maybe) software encoding. I'll refrain from posting more to this particular ticket. (I'm still lost though and IMHO discord's chat is not really adapted for technical discussions.)

[anancion]

Did you find a solution ? I've the exact same issue, with the latest version (v0.3.1) of the moonlight addon on libreelec. It launches and works just fine but performance is terrible. Cannot have a fluid video stream.

I tried

• server: PC (GeForce Experience), client: RPI4 Libreelec using a wireless mesh network, ethernet. Terrible performance.
• server: PC (GeForce Experience), client: Laptop running Linux mint on the same room as the RPI4, but over wifi. Perfect experience even with higher bitrate.
• server: said laptop running Sunshine same network condition, client: RPI4 Libreelec on ethernet. Terrible performance.

My guess is a performance bottleneck regarding the video decoding. But tweaking the /flash/config.txt file (tried different dtoverlay) didn't help.

Any idea ?

[wiwimouse]

I solved the problem by turning off the built-in SPI firewall on my WIFI

[gitandriu]

I solved the problem by turning off the built-in SPI firewall on my WIFI

Thanks for the inspiration. In my case it was Verizon's Wi-Fi QoS / WMM (supposedly "improves the quality of service (QoS) for voice, video, and audio streaming over Wi-Fi by prioritizing these data streams", but was doing just the opposite).