Support for v4l2loopback #2232
Support for v4l2loopback #2232
Conversation
The process API provides the system-specific implementation, the adb API uses it to expose adb commands.
Declare all the source files (including the platform-specific ones) at the beginning.
Adding "simple" in the function name brings no benefit.
On Linux, waitpid() both waits for the process to terminate and reaps it (closes its handle). On Windows, these actions are separated into WaitForSingleObject() and CloseHandle(). Expose these actions separately, so that it is possible to send a signal to a process while waiting for its termination without race condition. This allows to wait for server termination normally, but kill the process without race condition if it is not terminated after some delay.
It had been replaced by struct sc_port_range in scrcpy.h.
The header libavformat/version.h was included, but not libavcodec/version.h. As a consequence, the LIBAVCODEC_VERSION_INT definition depended on the caller includes.
The size, point and position structs were defined in common.h. Move them to coords.h so that common.h could be used for generic code to be included in all source files.
Include config.h and compat.h in common.h, and include common.h from all source files.
This enables necessary functions once for all. As a consequence, define common.h before any other header.
The function control_msg_serialize() returns a size_t.
The function process_wait() returned a bool (true if the process terminated successfully) and provided the exit code via an output parameter exit_code. But the returned value was always equivalent to exit_code == 0, so just return the exit code instead.
There were two versions: process_wait() and process_wait_noclose(). Expose a single version with a flag (it was already implemented that way internally).
The current process could be waited both by run_file_handler() and file_handler_stop(). To avoid the race condition, wait the process without closing, then close with mutex locked.
An "adb push" command is not terminated by SIGTERM.
Terminating the file handler current process may be either a "push" or "install" command.
Small unsigned integers promote to signed int. As a consequence, if v is
a uint8_t, then (v << 24) yields an int, so the left shift is undefined
if the MSB is 1.
Cast to uint32_t to yield an unsigned value.
Reported by USAN (meson x -Db_sanitize=undefined):
runtime error: left shift of 255 by 24 places cannot be represented
in type 'int'
The port_range is used from "struct server_params", the copy in "struct server" was unused.
Make strdup() available on all platforms.
The functions SDL_malloc(), SDL_free() and SDL_strdup() were used only because strdup() was not available everywhere. Now that it is available, use the native version of these functions.
The goal is to expose a consistent API for system tools, and paves the way to make the "core" independant of SDL in the future.
Add a function to assert that the mutex is held (or not).
There were only two frames simultaneously: - one used by the decoder; - one used by the renderer. When the decoder finished decoding a frame, it swapped it with the rendering frame. Adding a third frame provides several benefits: - the decoder do not have to wait for the renderer to release the mutex; - it simplifies the video_buffer API; - it makes the rendering frame valid until the next call to video_buffer_take_rendering_frame(), which will be useful for swscaling on window resize.
The flag is used only locally, there is no need to store it in the screen structure.
Video buffer is a tool between a frame producer and a frame consumer. For now, it is used between a decoder and a renderer, but in the future another instance might be used to swscale decoded frames.
As soon as the stream is started, the video buffer could notify a new frame available. In order to pass this event to the screen without race condition, the screen must be initialized before the screen is started.
Make the decoder independant of the SDL even mechanism, by making the consumer register a callback on the video_buffer.
A skipped frame is detected when the producer offers a frame while the current pending frame has not been consumed. However, the producer (in practice the decoder) is not interested in the fact that a frame has been skipped, only the consumer (the renderer) is. Therefore, notify frame skip via a consumer callback. This allows to manage the skipped and rendered frames count at the same place, and remove fps_counter from decoder.
Most of the fields are initialized dynamically.
The function screen_init_rendering had too many parameters.
Use a single function to initialize the screen instance.
During a frame swap, one of the two frames involved can be released.
The option is --encoder, not --encoder-name.
Virtual device is only for keyboard sources, not mouse or touchscreen
sources. Here is the value of InputDevice.getDevice(-1).toString():
Input Device -1: Virtual
Descriptor: ...
Generation: 2
Location: built-in
Keyboard Type: alphabetic
Has Vibrator: false
Has mic: false
Sources: 0x301 ( keyboard dpad )
InputDevice.getDeviceId() documentation says:
> An id of zero indicates that the event didn't come from a physical
> device and maps to the default keymap.
<https://developer.android.com/reference/android/view/InputEvent#getDeviceId()>
However, injecting events with a device id of 0 causes event.getDevice()
to be null on the client-side.
Commit 26529d3 used -1 as a workaround
to avoid a NPE on a specific Android TV device. But this is a bug in the
device system, which wrongly assumes that input device may not be null.
A similar issue was present in Flutter, but it is now fixed:
- <flutter/flutter#30665>
- <flutter/engine#7986>
On the other hand, using an id of -1 for touchscreen events (which is
invalid) causes issues for some apps:
<#2125 (comment)>
Therefore, use a device id of 0.
An alternative could be to find an existing device matching the source,
like "adb shell input" does. See getInputDeviceId():
<https://android.googlesource.com/platform/frameworks/base.git/+/master/cmds/input/src/com/android/commands/input/Input.java>
But it seems better to indicate that the event didn't come from a
physical device, and it would not solve #962 anyway, because an Android
TV has no touchscreen.
Refs #962 <#962>
Fixes #2125 <#2125>
BUTTON_PRIMARY must not be set for touch events: > This button constant is not set in response to simple touches with a > finger or stylus tip. The user must actually push a button. <https://developer.android.com/reference/android/view/MotionEvent#BUTTON_PRIMARY> Fixes #2169 <#2169>
|
Thank you for your work 👍 TestI managed to make it work on ./run d --v4l2sink /dev/video2 -NI play it with: vlc --network-caching=0 v4l2:///dev/video2However, there is sometimes a huge delay between the device and what is displayed in VLC. Especially at the beginning (seconds of delay), then a bit less but still a big delay. It a bit better if I limit the size ( BranchContributions should be based on Unfortunately, a lot of refactors/renaming occurred recently on Comments
No worries, at some point I will probably remove support for the old APIs anyway.
If the video is also displayed in a window, the stream will be decoded twice. Maybe it could be refactored to use the decoded frames only once (but it could be done later separately).
By default, BugAlso, at this point this is a detail, but if you pass an existing output device ( output(you can enable ASAN with |
|
Hi, thank you for reviewing this. I wasn't expecting a reply this fast. For VLC I experience the same, but I have the same problem with a capture card and another webcam, I think that VLC is not really meant for realtime capture. With Thank you for pointing out I had to use the dev branch. I was so exited to work on this that i completely forgot to check the obsvious. It's not a big problem, I did this project in a few hours, I think I'll manage to rebase it quickly. About your comments, I agree that the frame is decoded twice and I don't really like it either. In fact my first try was to share the video buffer to get the decoded frame. That was working very well when the video was displayed in a window but otherwise there was some logic to change around in screen.c and decoder.c regarding EVENT_NEW_FRAME and I didn't want to change too much outside my file, even considering the fact that this is only limited to linux. I can rework it to share the video buffer and decode the frame once if you prefer so. I will follow your suggestion about |
|
Please ignore this, I made a mess pushing my dev branch to master. I will create a new pull request. |
|
Merged into |
This request is to open a discussion about support for v4l2loopback that will also allows to support OBS on linux, as many have requested.
The code is working and it was tested with a few different phones and computers but other tests are more than welcome.
How to test
To test it first you'll need to
modprobe v4l2loopback, then runscrcpy --v4l2sink /dev/video0 -Nand open the loopback with some other program like VLC, ffplay or OBS (tested all 3 of them).The video stream can be simultaneously seen in scrcpy window, recorded with
--recordand sinked to v4l2loopback with--v4l2sink.It has not been tested without SCRCPY_LAVF_HAS_NEW_CODEC_PARAMS_API so there may be problems there.
How it works
All the magic is inside v4l2sink.c which is heavily based on decoder.c and recorder.c
In practice it uses v4l2enc from libavdevice as muxer in the same way that recorder.c use mp4 or matroska.
The key difference here is that v4l2loopback and most v4l2 clients expects to find a raw video frame and not an H264 encoded frame so like decoder.c it decodes the AVPacket to an AVFrame that is then re-encoded into an AVPacket with AV_CODEC_ID_RAWVIDEO.
The encoded packet is sent to v4l2enc and streamed to /dev/video0.
The code in v4l2sink.c and recorder.c are very similar and one can think that they can be merged together, but there are subtle differences here and there and for many reasons I prefer to keep them distinct.
RFC on known limitation
There is a limitation at the moment, it doesn't handle screen rotation, but it seems to be a limit of v4l2loopback where you can't change the resolution as long as anyone is using the device.
In practice if you start with the phone in portrait it will work only in portrait, if you start in landscape it will work only in landscape. To change orientation you have to close both scrcpy and the receiving application, rotate the phone and restart. Closing only one of them doesn't work.
Unless someone else come up with an idea to work around this I see two ways of handling this.
Idea 1, scrcpy can rotate back the frame so at least the image will be visible and one can rotate in OBS or other programs without any problem.
Idea 2, scrcpy can copy the frame centered in a squared texture. Eg, suppose that the frame is 720x1280, scrcpy will send to v4l2loopback a 1280x1280 texture with the frame centered and correctly rotated.
In your opinion what is the best solution? I see pro and cons in both of them.