Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use FFmpeg instead of EmguCV #236

Closed
Speykious opened this issue Oct 1, 2021 · 5 comments
Closed

Use FFmpeg instead of EmguCV #236

Speykious opened this issue Oct 1, 2021 · 5 comments
Assignees
Labels
area:recognition priority:medium Bugs or Features that may require attention sooner or later.
Projects

Comments

@Speykious
Copy link

Currently, EmguCV is being used only to handle webcam input.
We've had various problems with runtimes not being in the right place and cameras not being detected.

Thus I propose that we use FFmpeg for that task.
I think that it will be much easier to deal with as we can just use it as a system-installed binary.
Not to mention that the library is LGPL which is just perfect for our use-case.

@LeNitrous
Copy link
Member

Please provide details on how you plan to implement it as well.

@Speykious
Copy link
Author

From what I understand, all we need is to get the raw bytes of the image from the webcam.

Then you can refer to https://trac.ffmpeg.org/wiki/Capture/Webcam.
It would be as simple as reading the stdout from a command along the lines of

# Windows
ffmpeg -f vfwcap -i $device_index -pix_fmt rgba -f rawvideo -
# Linux
ffmpeg -f v4l2 -i /dev/video$device_index -pix_fmt rgba -f rawvideo -

to take raw RGBA pixels from the desired webcam.
We can even introduce an -input_format argument in case a device supports several input formats, for example on my machine:

❯ v4l2-ctl -d 0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'MJPG' (Motion-JPEG, compressed)
                Size: Discrete 1280x720
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 640x360
                        Interval: Discrete 0.033s (30.000 fps)
        [1]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 1280x720
                        Interval: Discrete 0.100s (10.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 640x360
                        Interval: Discrete 0.033s (30.000 fps)

You can test that the ffmpeg command above that outputs raw pixels works on Windows with

ffmpeg -f vfwcap -i 0 -pix_fmt rgba -f rawvideo - | ffmpeg -f rawvideo -pix_fmt rgba -s $sizeofwebcamhere -i - camerout.mkv

then cancel the command after a bit and open camerout.mkv in a video player.
Change the device number if you have several of them.

@sr229 sr229 added area:recognition priority:medium Bugs or Features that may require attention sooner or later. labels Oct 3, 2021
@sr229 sr229 added this to To do in Encore via automation Oct 3, 2021
@sr229
Copy link
Member

sr229 commented Oct 3, 2021

per the Discord discussion, we're re-assigning this to Encore 1 instead since this is more of something we wanna fix as Encore 1 is a stable client.

@Speykious
Copy link
Author

Speykious commented Nov 4, 2021

As Vignette Debut has been released, I and @adryzz have begun to engage in that task. We will create a small and agnostic library using FFmpeg.Autogen, which is a one-to-one mapping of the FFmpeg C API in C#, as it is preferable to use the low level API rather than the command line tool.

The lack of documentation doesn't make this task easy, but thankfully, most of the important stuff has been figured out.

@Speykious Speykious moved this from To do to In Progress in Encore Nov 4, 2021
@Speykious
Copy link
Author

Yay! SeeShark exists now.

Encore automation moved this from In Progress to Done Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:recognition priority:medium Bugs or Features that may require attention sooner or later.
Projects
Development

No branches or pull requests

4 participants