Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce latency by 1 frame #646

Closed
wants to merge 6 commits into from

Conversation

@rom1v
Copy link
Collaborator

commented Jul 12, 2019

This PR contains unrelated commit after I pushed force it (I cannot change the base branch). The relevant commit is 63af7fb.

To packetize the H.264 raw stream, av_parser_parse2() (called by av_read_frame()) knows that it has received a full frame only after it has received some data for the next frame. As a consequence, the client always waited until the next frame before sending the current frame to the decoder!

On the device side, we know packets boundaries. To reduce latency, make the device always transmit the "frame meta" to packetize the stream manually (it was already implemented to send PTS, but only enabled on recording).

On the client side, replace av_read_frame() by manual packetizing and parsing.

https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay
https://trac.ffmpeg.org/ticket/3354


I'm opening a pull request to get more tests before merging (the changes are quite sensitive), to avoid regressions.

To confirm that it actually reduces the end-to-end latency by 1 frame, I created a video which increments a number at 30fps (every 33.3ms):

# requires ffmpeg and imagemagick
for i in {0..999}; do convert -size 400x400 xc:White -gravity Center -pointsize 200 -font DejaVu-Sans-Mono -annotate 0 $i $(printf '%03d' $i).png; done
ffmpeg -framerate 30 -pattern_type glob -i '*.png' -c:v libx264 -vf format=yuv420p 30fps.mp4

I run this video in VLC on Android, and start scrcpy.

I compared the current master with this PR for both 1080x1920 (just running scrcpy) and 448x800 (running scrcpy -m800).

1080x1920

on master

1080x1920_30fps_before

End-to-end latency is about 3 frames (~100ms).

on lowlatency

1080x1920_30fps_lowlatency

End-to-end latency is about 2 frames (~67ms).

Once a packet is received by the client, on average, the frame is rendered 12ms later.
(with "skip frames" disabled, i.e. --render-expired-frames)

448x800

on master

448x800_30fps_before

End-to-end latency is about 2 frames (~67ms).

on lowlatency

448x800_30fps_lowlatency

End-to-end latency is about 1 frame (~33ms).

Once a packet is received by the client, on average, the frame is rendered 8ms later.
(with "skip frames" disabled, i.e. -m800 --render-expired-frames)


For now, I preserved the compatibility with prebuilt server v1.9, so you can just:

git fetch origin && git checkout lowlatency

then build as usual.

For windows users, I built a binary (replace the one from v1.9): scrcpy.exe.
SHA-256: 6b8eae4eb4df811b6aa194320f00b51102cf3a254e432f353963863c82f362ea

Thank you for your feedbacks 馃槈

beango1 and others added some commits Jun 23, 2019

Add option --window-title
Add an option to set a custom window title.

Signed-off-by: Romain Vimont <rom@rom1v.com>
Merge pull request #614 from beango1:window-title-simple
add option window-title to set the title

@rom1v rom1v force-pushed the lowlatency branch from 31c8799 to 84edcd6 Jul 12, 2019

@rom1v rom1v force-pushed the lowlatency branch 2 times, most recently from 0f0ccf4 to 0aa9828 Jul 29, 2019

rom1v added some commits Jul 30, 2019

Add option to change the push target
A drag & drop always pushed the file to /sdcard/.

Add an option to customize the target directory.

Fixes <#659>
Reduce latency by 1 frame
To packetize the H.264 raw stream, av_parser_parse2() (called by
av_read_frame()) knows that it has received a full frame only after it
has received some data for the next frame. As a consequence, the client
always waited until the next frame before sending the current frame to
the decoder!

On the device side, we know packets boundaries. To reduce latency,
make the device always transmit the "frame meta" to packetize the stream
manually (it was already implemented to send PTS, but only enabled on
recording).

On the client side, replace av_read_frame() by manual packetizing and
parsing.

<https://stackoverflow.com/questions/50682518/replacing-av-read-frame-to-reduce-delay>
<https://trac.ffmpeg.org/ticket/3354>

@rom1v rom1v force-pushed the lowlatency branch from 0aa9828 to 63af7fb Jul 30, 2019

@rom1v

This comment has been minimized.

Copy link
Collaborator Author

commented Jul 31, 2019

I reworked it and merged in dev branch: 63af7fb.

Based on that, I also made the recorder asynchronous, so that it never delay the decoder on blocking I/O: 35d9185.

Tests welcome to find bugs before the next release 馃槈

@rom1v rom1v closed this Jul 31, 2019

@rom1v rom1v deleted the lowlatency branch Jul 31, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can鈥檛 perform that action at this time.