videoio/FFmpeg: added CV_16UC1 read/write support #22894

mshabunin · 2022-11-30T22:58:03Z

Partially resolves #10623
Replaces #12284 and #12290

VideoCapture: we can now disable BGR conversion using parameter for the FFmpeg backend and return image in original format, straight from decoder. However this feature has many limitations, for example it is not possible to return multiplanar images yet - new interface or additional processing is necessary for this. So I added a warning with some details which will be printed when this feature is enabled. Basically 8UC1 and 16UC1 can now be returned from VideoCapture in addition to BGR.
VideoWriter: we can set depth using parameter during object initialization. It can be 8U or 16U. In combination with isColor it gives us three supported formats: 8UC3, 8UC1 and 16UC1; and one unsupported: 16UC3. 16UC1 can be handled by the FFV1 codec.
Several debug and warning log messages have been added for convenience and safety
The new test writes a video and then reads it back, frames are not compared with the original, only format and count is verified. Unsupported format combination is also checked in this test which is not very good, but I believe is fine for now.
FFV1 codec/container has been added to generic ffmpeg and videoio tests (regular BGR)

Note: some other codecs had hidden support for 16UC1, but I did not test them (PNG and RAWVIDEO).

Note: originally I wanted to modify #12290 - add a test, fix minor issues, but it turned out that I had to rewrite 90% of changes made in that PR, so I decided to publish it as a brand new one.

buildworker:Custom=linux-4
build_image:Custom=ffmpeg-master

kallaballa · 2022-12-01T09:09:14Z

Is there a codec that supports CV_16UC1 that has also FFMPEG video acceleration support?

kallaballa · 2022-12-01T09:33:51Z

At least my Linux/Intel Tigerlake/Media Driver system doesn't have a straight-forward way.

vainfo -a 2>&1 | grep BPP | sort | uniq
                                             VA_RT_FORMAT_RGB32_10BPP
                                             VA_RT_FORMAT_YUV420_10BPP

mshabunin · 2022-12-01T10:14:19Z

Looks like libva does not support single channel 16 bit images, there is no such format defined at least: https://intel.github.io/libva/group__api__core.html#gaff38bc6c01b65c6e9d473e40bdde9df6
However it has 16-bit-per-pixel RGB which should be supported by JPEG encoder - VA_RT_FORMAT_RGB16 (I think it is RGB565). But I don't know whether it works and if it is possible to store depth like this.

kallaballa · 2022-12-01T10:16:39Z

Looks like libva does not support single channel 16 bit images, there is no such format defined at least: https://intel.github.io/libva/group__api__core.html#gaff38bc6c01b65c6e9d473e40bdde9df6 However it has 16-bit-per-pixel RGB which should be supported by JPEG encoder - VA_RT_FORMAT_RGB16 (I think it is RGB565). But I don't know whether it works and if it is possible to store depth like this.

You are right! I missed the VA_RT_FORMAT_RGB16 possibility for MPJEG. I am going try it.

kallaballa · 2022-12-01T15:28:46Z

Build fails for me: build.log
I configured like following:

ccmake -DCMAKE_BUILD_TYPE=Release -DOPENCV_FFMPEG_ENABLE_LIBAVDEVICE=ON -DWITH_OPENGL=ON -DWITH_QT=ON DWITH_FFMPEG=ON -DOPENCV_FFMPEG_SKIP_BUILD_CHECK=ON -DWITH_VA=ON -DWITH_VA_INTEL=ON -DBUILD_PERF_TESTS=OFF -DBUILD_TESTS=OFF -DBUILD_EXAMPLES=OFF ..

FFmpeg:

# tag 5.1.2
ffmpeg version N-109236-g2324b917fc Copyright (c) 2000-2022 the FFmpeg developers
built with gcc 12 (SUSE Linux)
configuration: --prefix=/usr/local/ --libdir=/usr/local/lib64/ --bindir=/usr/local/bin --disable-static --enable-shared --enable-vaapi --enable-libvorbis --enable-libvpx --enable-gpl --cpu=native --enable-opengl --enable-libpulse --enable-libfdk-aac --enable-libx264 --enable-libx265 --enable-nonfree --enable-libv4l2
libavutil      57. 43.100 / 57. 43.100
libavcodec     59. 54.100 / 59. 54.100
libavformat    59. 34.102 / 59. 34.102
libavdevice    59.  8.101 / 59.  8.101
libavfilter     8. 50.101 /  8. 50.101
libswscale      6.  8.112 /  6.  8.112
libswresample   4.  9.100 /  4.  9.100
libpostproc    56.  7.100 / 56.  7.100

Seems to be an FFmpeg compatibility issue. Which version of FFmpeg is recommended for 4.x?

kallaballa · 2022-12-01T17:09:40Z

I am rebuilding FFmpeg release/5.1 right now just to see if the problem is on my side.

mshabunin · 2022-12-01T17:13:42Z

I tested this patch with multiple FFmpeg versions including 5.1.2, it should work. Are you sure you are using tagged release version and not the latest master branch?
Here is my OpenCV cmake output:

--     FFMPEG:                      YES
--       avcodec:                   YES (59.37.100)
--       avformat:                  YES (59.27.100)
--       avutil:                    YES (57.28.100)
--       swscale:                   YES (6.7.100)
--       avresample:                NO
--       avdevice:                  YES (59.7.100)

-DOPENCV_FFMPEG_SKIP_BUILD_CHECK=ON

Perhaps you have a problem with FFmpeg detection. Try to uncomment this line and determine why this happens.

kallaballa · 2022-12-01T17:14:37Z

I am not sure. that is why i am rebuilding.

kallaballa · 2022-12-01T17:28:13Z

Problem is on my side. Sorry! About to test If i can make it read gray16 using VA_INTEL.

kallaballa · 2022-12-01T18:08:05Z

Hmm. creating ffv1/gray16le is easy but i still fail to create a mjpeg/rgb16 video with ffmpeg. but i don't give up yet. anyway great PR!

mshabunin · 2022-12-01T18:59:01Z

This PR does not add 16 bit support for MJPEG. You can try to do it manually here: https://github.com/mshabunin/opencv/blob/39ddf0e4e22536847580bc0d21e35c0c20ebe8b9/modules/videoio/src/cap_ffmpeg_impl.hpp#L2932
Note, that in this PR we treat 16U input as GRAY16LE pixel format, so you probably need to change it to RGB16 for MJPEG or something like this.

Also, I'm not sure what would happen with 16 bit values after conversion to RGB16 and lossy compression.

kallaballa · 2022-12-01T19:37:44Z

This PR does not add 16 bit support for MJPEG. You can try to do it manually here: https://github.com/mshabunin/opencv/blob/39ddf0e4e22536847580bc0d21e35c0c20ebe8b9/modules/videoio/src/cap_ffmpeg_impl.hpp#L2932 Note, that in this PR we treat 16U input as GRAY16LE pixel format, so you probably need to change it to RGB16 for MJPEG or something like this.

Also, I'm not sure what would happen with 16 bit values after conversion to RGB16 and lossy compression.

Thx. I do understand! At the moment I am experimenting with accelerating raw RGB16 using VAAPI.

kallaballa · 2022-12-01T21:14:40Z

Ok. Two things i can say:

FFmpeg simply doesn't support rawvideo VAAPI acceleration (while VAAPI does).
FFmpeg doesn't seem to support RGB16 (rgb565le) with MJPEG: Incompatible pixel format 'rgb565le' for codec 'mjpeg', auto-selecting format 'yuvj444p' (while VAAPI does)

I'll tinker a little more and will open an issue at FFmpeg trac.

asmorkalov · 2022-12-02T07:56:36Z

Related: #9123

alalek · 2022-12-03T15:09:06Z

modules/videoio/src/cap_ffmpeg_legacy_api.hpp

@@ -30,7 +30,7 @@ OPENCV_FFMPEG_API int cvSetCaptureProperty_FFMPEG(struct CvCapture_FFMPEG* cap,
 OPENCV_FFMPEG_API double cvGetCaptureProperty_FFMPEG(struct CvCapture_FFMPEG* cap, int prop);
 OPENCV_FFMPEG_API int cvGrabFrame_FFMPEG(struct CvCapture_FFMPEG* cap);
 OPENCV_FFMPEG_API int cvRetrieveFrame_FFMPEG(struct CvCapture_FFMPEG* capture, unsigned char** data,
-                                             int* step, int* width, int* height, int* cn);
+                                             int* step, int* width, int* height, int* cn, int* depth);


AFAIK, it is a public API.
We should not modify existed functions.

I thought we've switched to the new plugin API in the FFmpeg plugin? Is it possible, what do you think?

These functions are still exported.
These functions are deprecated and they will to be removed in 5.x. So no need trying to modify it.

I've added a new function.

alalek · 2022-12-03T15:10:09Z

modules/videoio/src/cap_ffmpeg_impl.hpp

@@ -556,7 +555,7 @@ struct CvCapture_FFMPEG
    double getProperty(int) const;
    bool setProperty(int, double);
    bool grabFrame();
-    bool retrieveFrame(int flag, unsigned char** data, int* step, int* width, int* height, int* cn);
+    bool retrieveFrame(int flag, unsigned char** data, int* step, int* width, int* height, int* cn, int* deptth);


deptth

typo

alalek

Thank you!
Please squash commits into one before merge.

kallaballa mentioned this pull request Dec 1, 2022

Depth Encoding FFMPEG 16bit file support #22872

Closed

asmorkalov added this to the 4.7.0 milestone Dec 1, 2022

asmorkalov added category: videoio future This issue can't be fixed during restrictions of the compatibility requirements labels Dec 1, 2022

alalek reviewed Dec 3, 2022

View reviewed changes

mshabunin force-pushed the ffmpeg-16bit branch from 39ddf0e to cf511c7 Compare December 6, 2022 18:19

alalek approved these changes Dec 6, 2022

View reviewed changes

videoio/FFmpeg: added CV_16UC1 read/write support

6ad2165

mshabunin force-pushed the ffmpeg-16bit branch from cf511c7 to 6ad2165 Compare December 7, 2022 09:14

opencv-pushbot merged commit 49f539c into opencv:4.x Dec 8, 2022

alalek mentioned this pull request Dec 8, 2022

videoio(test): test skip due to non-updated FFmpeg wrapper #22933

Merged

mshabunin deleted the ffmpeg-16bit branch December 8, 2022 17:56

alalek mentioned this pull request Dec 20, 2022

videoio: valgrind issue on ffmpeg_16bit test (2022-12-19) #22991

Closed

alalek mentioned this pull request Jan 8, 2023

(5.x) Merge 4.x #23113

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

videoio/FFmpeg: added CV_16UC1 read/write support #22894

videoio/FFmpeg: added CV_16UC1 read/write support #22894

mshabunin commented Nov 30, 2022 •

edited by alalek

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022 •

edited

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022 •

edited

kallaballa commented Dec 1, 2022 •

edited

kallaballa commented Dec 1, 2022

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022 •

edited

asmorkalov commented Dec 2, 2022

alalek Dec 3, 2022

mshabunin Dec 4, 2022

alalek Dec 6, 2022

mshabunin Dec 6, 2022

alalek Dec 3, 2022

alalek left a comment

videoio/FFmpeg: added CV_16UC1 read/write support #22894

videoio/FFmpeg: added CV_16UC1 read/write support #22894

Conversation

mshabunin commented Nov 30, 2022 • edited by alalek

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022 • edited

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022 • edited

kallaballa commented Dec 1, 2022 • edited

kallaballa commented Dec 1, 2022

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022

mshabunin commented Dec 1, 2022

kallaballa commented Dec 1, 2022

kallaballa commented Dec 1, 2022 • edited

asmorkalov commented Dec 2, 2022

alalek Dec 3, 2022

Choose a reason for hiding this comment

mshabunin Dec 4, 2022

Choose a reason for hiding this comment

alalek Dec 6, 2022

Choose a reason for hiding this comment

mshabunin Dec 6, 2022

Choose a reason for hiding this comment

alalek Dec 3, 2022

Choose a reason for hiding this comment

alalek left a comment

Choose a reason for hiding this comment

mshabunin commented Nov 30, 2022 •

edited by alalek

kallaballa commented Dec 1, 2022 •

edited

kallaballa commented Dec 1, 2022 •

edited

kallaballa commented Dec 1, 2022 •

edited

kallaballa commented Dec 1, 2022 •

edited