Quest 3 video feed over Wi-Fi "skips" at different times

[sorryaboutyourcats]

• I have read the FAQ.
• I have searched in existing issues.

Environment

• Windows 10 22H2
• 5900X | 3090
• SCRCPY 2.1.1
• Windows release
• Quest 3
• Android 12.1

Describe the bug
Video feed over Wi-Fi "skips" at different times [i.e. can be smooth for ~20 seconds, then it skips].

Footage
Video of the skip in action here. The first part compares the SCRCPY recording with the OBS recording of the live feed.
The second part shows the live feed recorded from my phone [without OBS]. logcat here.

Video used:
scrcpy -b 16M --crop 2064:2208:2064:0 --no-audio --print-fps --disable-screensaver --record 16.mp4

Info

• SCRCPY doesn't count it as a frame skip the majority of the time, but instead as a jump in FPS [e.g. 90,90,120,60,90,90]
• Both h264 encoders produced the same skips
• All h265 encoders had too much latency to be useable
• Tried max FPS 60 [default Q3 FPS @ 90]
• Reported skipped frames CAN occur while display buffer is active [tested up to 500ms]
• Tried on multiple different APs/routers at different locations [GT-AXE16000 | Archer BE550 | CR1000A]
• All 6GHz & 5GHz channels tested
• 2,400Mbps up/down speeds on 160 MHz [also tested with 80 MHz channel bandwidth]
• Tested on two different computers [both AMD machines]
• Tested if cropping or window size affected anything
• Tested using 4 different NICs [overkill, yes!]
• Shut off apps/services running in the background that would affect video [e.g. ShadowPlay, etc.]
• Virtual Desktop runs smooth with 48M using same equipment [can go even higher actually]
• Disconnected Wi-Fi on main router rooms away in case of conflicts
• Used LatencyMon during recording to see if any issues came up
• Tried different power source & power strip for AP in case of grounding issue
• AP has no other devices connected to it via Wi-Fi, including phone [it sees other devices on network though]
• PC is directly connected to AP via ethernet cable
• Ping times are low to connected devices with no time outs
• Tweaked all settings I could on the routers [without putting a custom firmware like Merlin]
• Disabled NICs that weren't being used and uninstalled NICs used for VMs [e.g. Hyper-V, VMware, etc.]
• Uninstalled a VPN from Brave that I didn't realize was installed [WTF Brave!?]
• Tried max size as little as 400 and still had skipping

Smooth when...

• Watching recorded footage via SCRCPY [but not the live feed]
• Directly connected via USB [tested up to 40M]
• Bitrate to 1M with a 500ms buffer
• Forcing Quest 3 to run @ 72Hz instead of 90 [skips less]

Last notes
Considering it works over USB, the problem looks to be my network.
However, I've tried so much already that I'm losing hope. 😹

I could try to capture another device, like my Fold 4 to see if the issue is the Quest 3.
I could also try SCRCPY on my laptop, which is Intel, to see if the problem lies with AMD [doubt it].

Greatly appreciate any help on this matter, and thank you for making such an awesome tool! 🙏

[rom1v]

Basically, it's due to network jitter/packet loss.

If a TCP network packet is delayed or lost, it is retransmitted (this adds latency and delays every following packets). In that case, the scrcpy client has to wait to receive a new frame, causing a stutter, then it typically receives a burst of frames at once, so it renders them as fast as possible (skipping some if it decodes fast enough, to always render the very last one).

If you add a --display-buffer (as you tested), this may absorb the jitter and avoid the problem.

This does not happen when recording with --record=, because recording is not impacted by network jitter: the packets are captured and timestamped on the device, even if they are transmitted irregularly, they are muxed with their original timestamps.

[sorryaboutyourcats]

Thank you for replying back so quickly! 😻

I did more testing over USB 3 and ended up seeing a similar issue [it stutters at times instead of skips].

Video here. First part has OBS recording on left, SCRCPY recording on right.
Second part starting at 0:33 has phone recording with no OBS active.

scrcpy -b 16M --crop 2064:2208:2064:0 --no-audio --print-fps --max-fps 90

I have yet to try SCRCPY on my phone, or a different USB port, but may have to test those as well. 🙃
Or I may have to research what settings & hardware others are using to capture with their Quest 3. 🤔

[rom1v]

OK, it may not be only wireless lag, but also some encoder lag, decoder lag, computer lag… Some frame arrive too late, so it stutters, but this does not impact recording.

[sorryaboutyourcats]

Ah yes, recording through scrcpy is fine, but I'd be using this for streaming to Twitch [sorry for not elaborating that part 😹].
I could use the built-in casting on the Quest 3 which doesn't skip, but my god the FPS are so low that it's painful to watch. 😫

I don't mind upgrading my system in order to reduce lag if that'll help - main system is a Ryzen 9 5900X with a 3090.

[rom1v]

If you reduce the frame rate (from 90 to 60 for example) and the resolution, it might limit the stuttering (it would less likely cause a system lag).

[rom1v]

Also, capturing the window via OBS might cause stuttering too (there are some common graphical resources that must be shared).

If you're on Linux, you should try exposing the video stream via v4l2 that you could capture directly within OBS (without window).

[sorryaboutyourcats]

I actually recorded a 60 fps test last night! It's like your in my mind. 😹 Just uploaded it here.
The video does jump between buttery smooth and choppy though. 😿 This was over Wi-Fi.

This was used:
scrcpy -b 16M --crop 2064:2208:2064:0 --no-audio --print-fps --max-fps 60 --max-size 1080 --display-buffer 100

The skips happen regardless of OBS being used as the second part of the two videos I sent prior have me recording it from my phone, with my system fully dedicated to scrcpy. all hail scrcpy 🙏.

[rom1v]

Is there stuttering with --display-buffer=500, or even --display-buffer=1000?

[sorryaboutyourcats]

No skipping with 1000ms buffer, however the choppiness still goes in and out.

OBS taking over GPU/CPU resources does make sense for the bouts of chop, so I recorded the feed from another computer as well [via capture card, not anything network related like NDI], though the choppiness remained. 😿

Footage here. Shorter video this time [and I do appreciate your help with this! 🙏]
First two parts recorded on main system, next two parts recorded from another system.

[sorryaboutyourcats]

I was able to resolve that choppy-feed issue by removing the FPS limiter. Sure it still skips every now and then even with a 500ms buffer, but it's way smoother overall. #nomorechop

scrcpy -b 16M --crop 2064:2208:2064:0 --print-fps --no-audio --display-buffer 500

And for anyone else trying to stream native Quest 3 [not PCVR] games, use a Rode's Wireless GO II for audio. 😸👍
[In OBS, make sure sync offset is at 500ms for audio feed.]

[KptnKMan]

@sorryaboutyourcats out of interest, how are you capturing the OBS feed on the left side of your video?
I've been looking for a similar solution myself.

Cheers.

[sorryaboutyourcats]

@KptnKMan Here's a brief version of my set up so far:

scrcpy batch [copy to .bat file, change IP!]

echo
mode 28,12
echo
adb devices
echo
echo TCP IP port to 5555
adb -d tcpip 5555
adb connect 192.168.1.193:5555
echo
echo Bye bye USB
pause
scrcpy -b 16M --crop 2064:2208:2064:0 --print-fps --no-audio --display-buffer 500 --max-size 1080

Plugin to remove warped image

https://obsproject.com/forum/resources/obs-shaderfilter.1736/

OBS setup to remove warped image

1. New Game Capture scene and select scrcpy
2. Go to Filters and select User-defined shader, then select fisheye-xy.shader
3. Change Power x & Power y to -0.31

Audio

I was unable to get audio working correctly with scrcpy on the Quest 3 [at least as of a few weeks ago] so I opted to use the Rode Wireless Go IIs. Keep in mind they are mono each, so you'd need both of them with a splitter to work.

OBS setup for audio sync

1. Open Advanced Audio Properties
2. Change Sync Offset to 500 ms

Fin

Let me know if you have any other questions! I didn't mention how to read chat, and that's a whole other thing. 😹
If a proper guide would help, I can work on it over this weekend.
Happy VRing. 😸

[KptnKMan]

@sorryaboutyourcats that's great thanks, I now have that fisheye filter working in OBS.

But I was wondering specifically, how did you get that full frame square, where it says "OBS Recording [Happens regardless of OBS]"?

Because you seem to have a full frame there even with the bottom left of the frame which is usually cutoff.

Are you combining something to fill in or is that recorded on the headset?

I have a similar scrcpy script written in PowerShell, that I wrote some time ago, which does pretty much the same thing and a few other bits. I've been struggling to get a full frame like that, with all corners.

How did you get that?

[sorryaboutyourcats]

@KptnKMan It's just recorded on the headset, nothing combined. Hopefully this helps! ⬇

So here is how it looks like with & without the Windows XP overlay:

Which is a rotation of -17.60° [which might need adjusting] with an overlay on top to make it look neat. 😹

Transform settings:

The other setting was in my batch, which is --crop 2064:2208:2064:0. Other settings I saw online cropped more.

Let me know if I missed something. 👍

[Mike-666-exe]

@sorryaboutyourcats @KptnKMan
I dont know if this helps at all, but incase it helps

ive managed to get great results for both a square image with the least amount of crop along with decent audio. - mileage may vary

The best luck I've managed to capture the audio is honestly by streaming via the internal streaming feature to my browser just for the audio. there's often a slight delay that I would fix this in davinci, but that audio buffer might help (the reason I'm here lol)

I have also used side quest to adjust my settings a little bit so avoid game lag. by clicking on the wrench icon.

• fps 60
• FFR off
  -chromatic ab off
• Tatiana preset on (if you didn't know widens the internal recording feature of the quest by alot.
  also isn't necessary here as its irrelevant but wanted to share anyway incase you prefer to record that way.
• full rate capture on to match the recording fps at 60. (again not a factor when recording with scrcoy)

I'm on a quest 2 and get an almost perfect square with
scrcpy --no-audio --crop=1920:1920:1950:0

Using the obs shader filter, I think ive almost perfected it but primarily via trial and error so im sure someone could do better
what i do is stack a second shader filter over it with the fish eye correction.
you can get a perfect square almost with one but with a narrow fov.
So the second one is to correct that tunnel focus to level it out like so

(my quest camera lens has this weird glitch on the upper left but its fine because the lens is taller that it is wider so i can crop the entire top off for a more squared out image. )

As you can see when I stack the filter with these settings what I end up with is a perfectly corrected image with all my vertical and horizontal straight edge 99.999 straight.
The only thing cropped out are the bottom far right and left corners vs 20% of the screen otherwise.

(if your lenses aren't glitchy like mine then you only have the top far corners cut out. none of the far top, top, left, right, is cut out leading to the fullest view you you can get aside from just 3 small corners.)

Heres the full crop added to tidy it up.