InferROI.TestStreamingInfer test hangs on CI in OpenVINO configuration

[asmorkalov]

System Information

Platform Linux

CI link example: https://github.com/opencv/opencv/actions/runs/7956450017/job/21717195340?pr=25048
Dockerfile: https://github.com/opencv-infrastructure/opencv-gha-dockerfile/blob/main/ubuntu-github-actions-openvino--20.04/Dockerfile
Image: quay.io/opencv-ci/opencv-ubuntu-20.04-openvino:20230724
CI pipeline: https://github.com/opencv/ci-gha-workflow/blob/main/.github/workflows/OCV-PR-4.x-U20-OpenVINO.yaml

Detailed description

Steps to reproduce

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[dmatveev]

Oh is it failing again?

[asmorkalov]

no. should i close the issue?

[asmorkalov]

Looks like the issue is still there, but become less often. Fresh example: https://github.com/opencv/opencv/actions/runs/8153782389/job/22285849425?pr=25149

[dmatveev]

@asmorkalov this is certainly not THAT test which hang constantly, but more like a floating issue I reported over email during my early investigation (hang in the VideoCapture itself). I believe this test follows the same pattern where the VideoCapture issue reproduces. Please open a new issue as the originally reported InferROI.TestStreamingInfer is no longer there - and I will post my past findings there.

[dkurt]

@dmatveev, are yours observations related to some memory issues? I see that there is a valgring issue on videoio tests: https://pullrequest.opencv.org/buildbot/builders/4_x_valgrind-lin64-debug/builds/100068/steps/test_videoio/logs/stdio

[ RUN      ] videoio/videoio_synthetic.write_read_position/7, where GetParam() = (640x480, FOURCC(FFV1), .avi, FFMPEG, 30dB)
==17791== Invalid read of size 8
==17791==    at 0xAA6876F: ??? (in /usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100)
==17791==    by 0xAA5E6A1: ??? (in /usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100)
==17791==    by 0xAA522BE: sws_scale (in /usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100)
==17791==    by 0x50F3488: CvVideoWriter_FFMPEG::writeFrame(unsigned char const*, int, int, int, int, int) (cap_ffmpeg_impl.hpp:2546)
==17791==    by 0x50F37AC: cvWriteFrame_FFMPEG (cap_ffmpeg_impl.hpp:3377)
==17791==    by 0x50F38A8: cv::(anonymous namespace)::CvVideoWriter_FFMPEG_proxy::write(cv::_InputArray const&) (cap_ffmpeg.cpp:178)
==17791==    by 0x5081436: cv::VideoWriter::write(cv::_InputArray const&) (cap.cpp:738)
==17791==    by 0x5081558: cv::VideoWriter::operator<<(cv::Mat const&) (cap.cpp:746)
==17791==    by 0x1C8C45: opencv_test::videoio_synthetic::writeVideo() (test_video_io.cpp:301)
==17791==    by 0x1C443F: opencv_test::Videoio_Test_Base::doTest() (test_video_io.cpp:50)
==17791==    by 0x1C5324: opencv_test::videoio_synthetic_write_read_position_Test::Body() (test_video_io.cpp:438)
==17791==    by 0x1B1B28: opencv_test::videoio_synthetic_write_read_position_Test::TestBody() (test_video_io.cpp:438)
==17791==  Address 0x3e42307d is 921,597 bytes inside a block of size 921,600 alloc'd
==17791==    at 0x4C33E76: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17791==    by 0x4C33F91: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==17791==    by 0x5CF24CD: cv::fastMalloc(unsigned long) (alloc.cpp:137)
==17791==    by 0x5DE1CA0: cv::StdMatAllocator::allocate(int, int const*, int, void*, unsigned long*, cv::AccessFlag, cv::UMatUsageFlags) const (matrix.cpp:147)
==17791==    by 0x5DE5280: cv::Mat::create(int, int const*, int) (matrix.cpp:703)
==17791==    by 0x5DE5EE4: cv::Mat::create(int, int, int) (matrix.cpp:533)
==17791==    by 0x5DE6256: cv::Mat::Mat(cv::Size_<int>, int) (matrix.cpp:360)
==17791==    by 0x1C8A5A: opencv_test::videoio_synthetic::writeVideo() (test_video_io.cpp:294)
==17791==    by 0x1C443F: opencv_test::Videoio_Test_Base::doTest() (test_video_io.cpp:50)
==17791==    by 0x1C5324: opencv_test::videoio_synthetic_write_read_position_Test::Body() (test_video_io.cpp:438)
==17791==    by 0x1B1B28: opencv_test::videoio_synthetic_write_read_position_Test::TestBody() (test_video_io.cpp:438)
==17791==    by 0x1F706B: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (ts_gtest.cpp:3919)
==17791==    by 0x1F706B: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (ts_gtest.cpp:3955)
==17791== 
{
   <insert_a_suppression_name_here>
   Memcheck:Addr8
   obj:/usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100
   obj:/usr/lib/x86_64-linux-gnu/libswscale.so.4.8.100
   fun:sws_scale
   fun:_ZN20CvVideoWriter_FFMPEG10writeFrameEPKhiiiii
   fun:cvWriteFrame_FFMPEG
   fun:_ZN2cv12_GLOBAL__N_126CvVideoWriter_FFMPEG_proxy5writeERKNS_11_InputArrayE
   fun:_ZN2cv11VideoWriter5writeERKNS_11_InputArrayE
   fun:_ZN2cv11VideoWriterlsERKNS_3MatE
   fun:_ZN11opencv_test17videoio_synthetic10writeVideoEv
   fun:_ZN11opencv_test17Videoio_Test_Base6doTestEv
   fun:_ZN11opencv_test42videoio_synthetic_write_read_position_Test4BodyEv
   fun:_ZN11opencv_test42videoio_synthetic_write_read_position_Test8TestBodyEv
}
[       OK ] videoio/videoio_synthetic.write_read_position/7 (84044 ms)

related code and workaround for similar problem:

opencv/modules/videoio/src/cap_ffmpeg_impl.hpp

Lines 2484 to 2489 in 2b1c8aa

	// FFmpeg contains SIMD optimizations which can sometimes read data past
	// the supplied input buffer.
	// Related info: https://trac.ffmpeg.org/ticket/6763
	// 1. To ensure that doesn't happen, we pad the step to a multiple of 32
	// (that's the minimal alignment for which Valgrind doesn't raise any warnings).
	// 2. (dataend - SIMD_SIZE) and (dataend + SIMD_SIZE) is from the same 4k page

PR: #6281

[dmatveev]

@dkurt my observations were related to GStreamer CAP_BACKEND actually, and the issue happened in its constructor. Still, I propose to close the current issue as it is not relevant anymore - this particular test was "fixed"

It seems there's a common issue with concurrent work of multiple VideoCaptures which we have in our Streaming tests so we need to look there. If there's no such issue yet, I can create one and put my findings there.