Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce latency by 1 frame #646

Closed
wants to merge 6 commits into from
Closed

Reduce latency by 1 frame #646

wants to merge 6 commits into from

Conversation

rom1v
Copy link
Collaborator

@rom1v rom1v commented Jul 12, 2019

This PR contains unrelated commit after I pushed force it (I cannot change the base branch). The relevant commit is 63af7fb.

To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder!

On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording).

On the client side, replace av_read_frame() by manual packetizing and parsing.

https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay
https://trac.ffmpeg.org/ticket/3354


I'm opening a pull request to get more tests before merging (the changes are quite sensitive), to avoid regressions.

To confirm that it actually reduces the end-to-end latency by 1 frame, I created a video which increments a number at 30fps (every 33.3ms):

# requires ffmpeg and imagemagick
for i in {0..999}; do convert -size 400x400 xc:White -gravity Center -pointsize 200 -font DejaVu-Sans-Mono -annotate 0 $i $(printf '%03d' $i).png; done
ffmpeg -framerate 30 -pattern_type glob -i '*.png' -c:v libx264 -vf format=yuv420p 30fps.mp4

I run this video in VLC on Android, and start scrcpy.

I compared the current master with this PR for both 1080x1920 (just running scrcpy) and 448x800 (running scrcpy -m800).

1080x1920

on master

1080x1920_30fps_before

End-to-end latency is about 3 frames (~100ms).

on lowlatency

1080x1920_30fps_lowlatency

End-to-end latency is about 2 frames (~67ms).

Once a packet is received by the client, on average, the frame is rendered 12ms later.
(with "skip frames" disabled, i.e. --render-expired-frames)

448x800

on master

448x800_30fps_before

End-to-end latency is about 2 frames (~67ms).

on lowlatency

448x800_30fps_lowlatency

End-to-end latency is about 1 frame (~33ms).

Once a packet is received by the client, on average, the frame is rendered 8ms later.
(with "skip frames" disabled, i.e. -m800 --render-expired-frames)


For now, I preserved the compatibility with prebuilt server v1.9, so you can just:

git fetch origin && git checkout lowlatency

then build as usual.

For windows users, I built a binary (replace the one from v1.9): scrcpy.exe.
SHA-256: 6b8eae4eb4df811b6aa194320f00b51102cf3a254e432f353963863c82f362ea

Thank you for your feedbacks 馃槈

beango1 and others added 3 commits June 24, 2019 19:58
Add an option to set a custom window title.

Signed-off-by: Romain Vimont <rom@rom1v.com>
add option window-title to set the title
A drag & drop always pushed the file to /sdcard/.

Add an option to customize the target directory.

Fixes <#659>
To packetize the H.264 raw stream, av_parser_parse2() (called by
av_read_frame()) knows that it has received a full frame only after it
has received some data for the next frame. As a consequence, the client
always waited until the next frame before sending the current frame to
the decoder!

On the device side, we know packets boundaries. To reduce latency,
make the device always transmit the "frame meta" to packetize the stream
manually (it was already implemented to send PTS, but only enabled on
recording).

On the client side, replace av_read_frame() by manual packetizing and
parsing.

<https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay>
<https://trac.ffmpeg.org/ticket/3354>
@rom1v
Copy link
Collaborator Author

rom1v commented Jul 31, 2019

I reworked it and merged in dev branch: 63af7fb.

Based on that, I also made the recorder asynchronous, so that it never delay the decoder on blocking I/O: 35d9185.

Tests welcome to find bugs before the next release 馃槈

@ghost
Copy link

ghost commented Jan 13, 2021

Tried with scrcpy --bit-rate 1M --max-size 240 --max-fps 30 got 2 frame latency at 136x240, It's my device dependent, if it was faster than rom1v's unknown device, it would be 0 to 1 frame latency, without scrcpy, I would struggle with compressed image, inconsistent frame rate and image freeze spikes, thanks that scrcpy existed!!!!!! 馃槆
1063

@rom1v
Copy link
Collaborator Author

rom1v commented Jan 14, 2021

Tried with scrcpy --bit-rate 1M --max-size 240 --max-fps 30 got 2 frame latency at 136x240, It's my device dependent,

Wow, that's a lot of latency for such a small definition.

if it was faster than rom1v's unknown device

The device I used in this test was a Nexus 5. The results are better with newer devices (like a OnePlus 7 Pro).

@ghost
Copy link

ghost commented Apr 28, 2023

Actually x265 outperforms in everything x264:
0 ms latency x265, 12ms latency x264. x264 just sucks. Native 2400x1080 (the phone's screen is left one, the images are dark because it is 1/12000s so numbers don't look blurry)
x264:
image

x265:
image

@rom1v
Copy link
Collaborator Author

rom1v commented Apr 28, 2023

I'm not sure to understand the screenshots, but it depends a lot on the device and hardware encoder. For example, on a One Plus 7 Pro, the h264 encoder has a lower latency than the h265 encoder.

@ghost
Copy link

ghost commented Apr 29, 2023

I'm not sure to understand the screenshots, but it depends a lot on the device and hardware encoder. For example, on a One Plus 7 Pro, the h264 encoder has a lower latency than the h265 encoder.

Basically i wrote a c++ program to display "i+1" number every 1ms, so i can take a photo and see how different are numbers on my phone and on my pc, basically latency.
With screen overclock to 77 hz (i wish there was a mod to trick android into rendering 120fps so the picture with scrcpy would be smoother), the frame time should be around 12ms, that's the latency of h264, however, h265 doesn't have any latency at all because numbers are the same (i mean of course it does, but the latency is smaller than 12ms)

It would be so much better, if h265 was the default one.

UPD: i just made a super slow motion video with both screens, it appears, that the latency is around 8-9ms, while using h265

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants