`cudacodec::VideoWriter` add container output #3569

cudawarped · 2023-10-05T13:37:11Z

Where supported, allows the raw encoded video output from Nvidia's hardware decoder to be encapsulated in a suitable (mp4/mkv/etc.) container.

Dependant on opencv/opencv#24363.

Fixes #3556

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the original bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

modules/cudacodec/src/video_writer.cpp

…ncapsulation videoio: Add raw encoded video stream muxing to cv::VideoWriter with CAP_FFMPEG #24363 Allow raw encoded video streams (e.g. h264[5]) to be encapsulated by `cv::VideoWriter` to video containers (e.g. mp4/mkv). Operates in a similar way to #15290 where encapsulation is enabled by setting the `VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO` flag when constructing `cv::VideoWriter` e.g. ``` VideoWriter container(fileNameOut, api, fourcc, fps, { width, height }, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1 }); ``` and each raw encoded frame is passed as single row of a `CV_8U` `cv::Mat`. The main reason for this PR is to allow `cudacodec::VideoWriter` to output its encoded streams to a suitable container, see opencv/opencv_contrib#3569. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

asmorkalov · 2023-10-25T10:48:24Z

@cudawarped Thanks a lot for the contribution! I merged related patch to main repo and re-triggered CI. Locally I get build error:

/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:408:102: error: invalid new-expression of abstract class type ‘cv::cudacodec::FFmpegVideoWriter’
             encoderCallback = new FFmpegVideoWriter(fileName, codec, fps, frameSize, params.idrPeriod);
                                                                                                      ^
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:70:7: note:   because the following virtual functions are pure within ‘cv::cudacodec::FFmpegVideoWriter’:
 class FFmpegVideoWriter : public EncoderCallback
       ^~~~~~~~~~~~~~~~~
In file included from /home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/precomp.hpp:54:0,
                 from /home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:44:
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/include/opencv2/cudacodec.hpp:212:18: note:    virtual void cv::cudacodec::EncoderCallback::onEncoded(std::vector<std::vector<unsigned char> >)
     virtual void onEncoded(std::vector<std::vector<uint8_t>> vPacket) = 0;
                  ^~~~~~~~~
modules/cudacodec/CMakeFiles/opencv_cudacodec.dir/build.make:229: recipe for target 'modules/cudacodec/CMakeFiles/opencv_cudacodec.dir/src/video_writer.cpp.o' failed

cudawarped · 2023-10-25T10:59:17Z

@asmorkalov Looks like i didn't add the const from your last review to the EncoderCallback Interface. Addressing now.

cudawarped · 2023-10-25T11:02:17Z

@asmorkalov Should I wait to see if the CI finishes successfuly or push that change now?

I waited all checks were successful pushing change now for you to test.

Note: The h264 and vp9 tests inside cudacodec currently fail on FFMpeg 2.8.17 see below. Is this an issue in the contrib repo?

[ FAILED ] 3 tests, listed below:
[ FAILED ] CUDA_Codec/Video.Reader/7, where GetParam() = (NVIDIA GeForce RTX 3070 Ti Laptop GPU, "highgui/video/sample_322x242_15frames.yuv420p.libvpx-vp9.mp4")
[ FAILED ] CUDA_Codec/EncoderParams.Writer/0, where GetParam() = (NVIDIA GeForce RTX 3070 Ti Laptop GPU, 5)
[ FAILED ] CUDA_Codec/EncoderParams.Writer/1, where GetParam() = (NVIDIA GeForce RTX 3070 Ti Laptop GPU, 10)

modules/cudacodec/test/test_video.cpp

asmorkalov · 2023-10-26T06:31:50Z

modules/cudacodec/src/video_writer.cpp

+FFmpegVideoWriter::FFmpegVideoWriter(const String& fileName, const Codec codec, const int fps, const Size sz, const int idrPeriod) {
+    if (!videoio_registry::hasBackend(CAP_FFMPEG))
+        CV_Error(Error::StsNotImplemented, "FFmpeg backend not found");
+    const int fourcc = codec == Codec::H264 ? cv::VideoWriter::fourcc('a', 'v', 'c', '1') : cv::VideoWriter::fourcc('h', 'e', 'v', '1');


What about VP8, VP9, MPEG4, others?

The Nvidia Video Codec SDK only supports H264, H265 and AV1 encoding, with cudacodec::VideoWriter only supporting H264[5]. cudacodec::VideoWriter was introduced/fixed/updated before Video Codec SDK v12.0 where AV1 encoding was introduced. Because AV1 is only supported on Ada GPU's and I don't have access to one for testing I have not yet added that functionality.

I will add comments to the create methods to clarify this.

Sorry, I screwed up with decoder. Please add assertion for codec then.

This is taken care of in CodecGuid which is called during the initialization of the VideoWriterImpl.

Just noticed the error message uses the incorrect codec identifiers I can fix this once the CI finishes.

modules/cudacodec/include/opencv2/cudacodec.hpp

asmorkalov · 2023-10-26T15:03:19Z

/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/NvEncoder.h:266:86: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     void* GetCompletionEvent(uint32_t eventIdx) { return (m_vpCompletionEvent.size() == m_nEncoderBuffer) ? m_vpCompletionEvent[eventIdx] : nullptr; }
                                                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp: In constructor ‘cv::cudacodec::FFmpegVideoWriter::FFmpegVideoWriter(const String&, cv::cudacodec::Codec, int, cv::Size, int)’:
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:69: error: ‘VIDEOWRITER_PROP_RAW_VIDEO’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:123: error: ‘VIDEOWRITER_PROP_KEY_INTERVAL’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                                                                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:165: error: no matching function for call to ‘cv::VideoWriter::open(const String&, const int&, const int&, const Size&, <brace-enclosed initializer list>)’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });

cudawarped · 2023-10-26T15:09:30Z

/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/NvEncoder.h:266:86: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
void* GetCompletionEvent(uint32_t eventIdx) { return (m_vpCompletionEvent.size() == m_nEncoderBuffer) ? m_vpCompletionEvent[eventIdx] : nullptr; }

#3517 will fix this.

cudawarped · 2023-10-26T15:13:19Z

/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp: In constructor ‘cv::cudacodec::FFmpegVideoWriter::FFmpegVideoWriter(const String&, cv::cudacodec::Codec, int, cv::Size, int)’:
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:69: error: ‘VIDEOWRITER_PROP_RAW_VIDEO’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:123: error: ‘VIDEOWRITER_PROP_KEY_INTERVAL’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                                                                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:165: error: no matching function for call to ‘cv::VideoWriter::open(const String&, const int&, const int&, const Size&, <brace-enclosed initializer list>)’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });

Not sure on these checking now.

cudawarped · 2023-10-26T15:41:37Z

/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp: In constructor ‘cv::cudacodec::FFmpegVideoWriter::FFmpegVideoWriter(const String&, cv::cudacodec::Codec, int, cv::Size, int)’:
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:69: error: ‘VIDEOWRITER_PROP_RAW_VIDEO’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:123: error: ‘VIDEOWRITER_PROP_KEY_INTERVAL’ is not a member of ‘cv::VideoWriterProperties’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });
                                                                                                                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/alexander/Projects/OpenCV/opencv_contrib/modules/cudacodec/src/video_writer.cpp:85:165: error: no matching function for call to ‘cv::VideoWriter::open(const String&, const int&, const int&, const Size&, <brace-enclosed initializer list>)’
     writer.open(fileName, fourcc, fps, sz, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1, VideoWriterProperties::VIDEOWRITER_PROP_KEY_INTERVAL, idrPeriod });

Not sure on these checking now.

You haven't updated your local main repo to include opencv/opencv#24363

asmorkalov

👍 Thanks a lot for the contribution!

…ream_encapsulation videoio: Add raw encoded video stream muxing to cv::VideoWriter with CAP_FFMPEG opencv#24363 Allow raw encoded video streams (e.g. h264[5]) to be encapsulated by `cv::VideoWriter` to video containers (e.g. mp4/mkv). Operates in a similar way to opencv#15290 where encapsulation is enabled by setting the `VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO` flag when constructing `cv::VideoWriter` e.g. ``` VideoWriter container(fileNameOut, api, fourcc, fps, { width, height }, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1 }); ``` and each raw encoded frame is passed as single row of a `CV_8U` `cv::Mat`. The main reason for this PR is to allow `cudacodec::VideoWriter` to output its encoded streams to a suitable container, see opencv/opencv_contrib#3569. ### Pull Request Readiness Checklist See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request - [x] I agree to contribute to the project under Apache 2 License. - [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV - [x] The PR is proposed to the proper branch - [x] There is a reference to the original bug report and related work - [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable Patch to opencv_extra has the same branch name. - [x] The feature is well documented and sample code can be built with the project CMake

cudawarped mentioned this pull request Oct 5, 2023

videoio: Add raw encoded video stream encapsulation to cv::VideoWriter with CAP_FFMPEG opencv/opencv#24363

Merged

6 tasks

asmorkalov reviewed Oct 18, 2023

View reviewed changes

modules/cudacodec/src/video_writer.cpp Outdated Show resolved Hide resolved

modules/cudacodec/src/video_writer.cpp Outdated Show resolved Hide resolved

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from 57f18a9 to a927d42 Compare October 19, 2023 13:27

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch 2 times, most recently from 53dbde7 to 5dd8bb5 Compare October 25, 2023 12:29

asmorkalov added the category: cuda label Oct 25, 2023

asmorkalov reviewed Oct 26, 2023

View reviewed changes

asmorkalov self-assigned this Oct 26, 2023

cudacodec: VideoWriter add container output

f40bc0c

cudawarped force-pushed the videoio_ffmpeg_add_stream_encapsulation branch from 5dd8bb5 to f40bc0c Compare October 26, 2023 10:39

asmorkalov approved these changes Oct 26, 2023

View reviewed changes

asmorkalov merged commit 1358800 into opencv:4.x Oct 26, 2023
9 checks passed

asmorkalov mentioned this pull request Nov 2, 2023

5.x merge 4.x #3587

Merged

cudawarped mentioned this pull request Dec 30, 2023

cudacodec: Enable features available with updated ffmpeg dll #3615

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`cudacodec::VideoWriter` add container output #3569

`cudacodec::VideoWriter` add container output #3569

cudawarped commented Oct 5, 2023 •

edited

Loading

asmorkalov commented Oct 25, 2023

cudawarped commented Oct 25, 2023

cudawarped commented Oct 25, 2023 •

edited

Loading

asmorkalov Oct 26, 2023

cudawarped Oct 26, 2023 •

edited

Loading

asmorkalov Oct 26, 2023

cudawarped Oct 26, 2023 •

edited

Loading

asmorkalov commented Oct 26, 2023

cudawarped commented Oct 26, 2023 •

edited

Loading

cudawarped commented Oct 26, 2023 •

edited

Loading

cudawarped commented Oct 26, 2023

asmorkalov left a comment

cudacodec::VideoWriter add container output #3569

cudacodec::VideoWriter add container output #3569

Conversation

cudawarped commented Oct 5, 2023 • edited Loading

Pull Request Readiness Checklist

asmorkalov commented Oct 25, 2023

cudawarped commented Oct 25, 2023

cudawarped commented Oct 25, 2023 • edited Loading

asmorkalov Oct 26, 2023

Choose a reason for hiding this comment

cudawarped Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

asmorkalov Oct 26, 2023

Choose a reason for hiding this comment

cudawarped Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

asmorkalov commented Oct 26, 2023

cudawarped commented Oct 26, 2023 • edited Loading

cudawarped commented Oct 26, 2023 • edited Loading

cudawarped commented Oct 26, 2023

asmorkalov left a comment

Choose a reason for hiding this comment

`cudacodec::VideoWriter` add container output #3569

`cudacodec::VideoWriter` add container output #3569

cudawarped commented Oct 5, 2023 •

edited

Loading

cudawarped commented Oct 25, 2023 •

edited

Loading

cudawarped Oct 26, 2023 •

edited

Loading

cudawarped Oct 26, 2023 •

edited

Loading

cudawarped commented Oct 26, 2023 •

edited

Loading

cudawarped commented Oct 26, 2023 •

edited

Loading