Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5.x merge 4.x #24862

Merged
merged 246 commits into from
Jan 22, 2024
Merged

5.x merge 4.x #24862

merged 246 commits into from
Jan 22, 2024

Conversation

asmorkalov
Copy link
Contributor

@asmorkalov asmorkalov commented Jan 15, 2024

The PR includes all changes in 4.x till 4.9.0 release.

OpenCV Contrib PR: opencv/opencv_contrib#3621
OpenCV Extra PR: opencv/opencv_extra#1142
Requires CI update in: opencv/ci-gha-workflow#134

#23021 from cudawarped:add_cuda_first_class_support
#23654 from zihaomu:wino_fp16
#24060 from TolyaTalamanov:at/advanced-device-selection-onnxrt-directml
#24068 from TolyaTalamanov:at/add-onnx-coreml-execution-provider
#24136 from komakai:visionos_support
#24205 from PeterJohnson:fix-msvc-arm64
#24224 from AsyaPronina:asyadev/port_vas_ot_to_opencv
#24243 from kecsap:4.x
#24271 from Kumataro:fix24163
#24299 from dkurt:qrcode_decode
#24364 from bagelbytes61:bugfix/qrcode-version-estimator
#24378 from fengyuentau:instance_norm
#24384 from Dhanwanth1803:feat-crop
#24396 from Tsai-chia-hsiang:yolov8cv
#24397 from richard28039:add_fcnresnet101_to_dnn_sample
#24445 from Abdurrahheem:ash/dev_einsum_pref
#24456 from alexlyulkov:al/aar
#24458 from laolaolulu:4.x
#24462 from fengyuentau:cann_support_hardswish
#24463 from dkurt:dnn_shared_nodes_fusion
#24465 from ivashmak:fix_usac_tutorial
#24466 WanliZhong:refactor_softmax
#24473 from alexlyulkov:al/samples_with_maven
#24474 from asmorkalov:as/BOWImgDescriptorExtractoor_java_ctor
#24476 from fengyuentau:attention_layer
#24477 from asmorkalov:as/SimpleBlobDetector_js
#24479 from AleksandrPanov:fix_contour_filtering
#24480 from savuor:backport_patch_nans
#24483 from dkurt:dnn_fusion_commutative_ops
#24485 from hanliutong:rvv-opt
#24488 from hipudding:npu_support
#24492 from asmorkalov:as/reprioritize_fps_source_ffmpeg
#24495 from vrabaud:fast_math_compile
#24496 from cabelo:yolov3
#24499 from ivashmak:usac_bug_fix
#24500 from fengyuentau:test_layer_fusion
#24509 from Abdurrahheem:ash/dev_einsum_fast_gemm
#24510 from asmorkalov:as/softmax_rvv
#24511 from asmorkalov:as/remove_rvv_internals_from_docs
#24512 from asmorkalov:as/gstreamer_warn_fix
#24519 from shubh1176:4.x
#24521 from dkurt:fix_broken_urls
#24526 from thewoz:arrow
#24527 from MaximSmolskiy:fix-out-of-image-corners-in-cv-cornersubpix
#24531 from fengyuentau:instancenorm_onnx_test_cuda
#24533 from AleksandrPanov:aruco_remove_filter_fix
#24535 from dkurt:ipp_distransform_update
#24539 from LaurentBerger:blobrecttoimage
#24544 from fengyuentau:layernorm_conformance
#24546 from thewoz:checkerboard
#24547 from WanliZhong:refactor_conv_perf_test
#24552 from fengyuentau:layernorm_backends
#24553 from skycat8:yolov5
#24554 from asmorkalov:as/android_sdk_local_props
#24560 from asmorkalov:as/openni_status
#24565 from CNClareChen:4.x
#24566 from MaximSmolskiy:fix-typos-in-calibinit.cpp
#24569 from Abdurrahheem:ash/padding_value_fix
#24575 from asmorkalov:as/drop_android_engine
#24576 from AsyaPronina:ot_to_python
#24577 from dkurt:dnn_graph_match_stack
#24581 from vrabaud:tsan
#24582 from hanliutong:rvv-lut
#24584 from TolyaTalamanov:at/implement-inference-only-mode-for-ov-backend
#24592 from savuor:recorder_android
#24595 from MaximSmolskiy:fix-typo-inChessBoardDetector-generateQuads
#24597 from MaximSmolskiy:fix-bug-in-ChessBoardDetector-findQuadNeighbors
#24598 from AleksandrPanov:add_android_qr_sample
#24599 from asmorkalov:as/android_face_detect_dnn
#24600 from asmorkalov:as/tutorial2_fix
#24601 from TolyaTalamanov:at/fix-onnxrt-include-path-in-cmake
#24604 from MaximSmolskiy:fix-time-measurement-in-test-Calib3d_ChessboardDetector.timing
#24605 from MaximSmolskiy:speed-up-ChessBoardDetector-findQuadNeighbors
#24607 from asmorkalov:as/serialize_android_samples_build
#24608 from spacegaier:patch-1
#24610 from jimmylaw21:dnn-onnx-add-group-norm-layer
#24611 from Abdurrahheem:ash/add_yolov6_test
#24613 from WanliZhong:softmax_default_axis
#24615 from smirnov-alexey:as/infer2_assert_soften
#24616 from asmorkalov:as/android_mobilenet_refresh
#24618 from vrabaud:compilation
#24619 from philnelson:patch-1
#24622 from alexlyulkov:al/fixed-android-sample-img-save
#24625 from asmorkalov:as/mjpeg_encoder_status
#24629 from Kumataro:add_crosscompile_with_multiarch
#24630 from tomoaki0705:fix_typo_calib3d
#24634 from jubinchheda:deprecated-ios-api-patches
#24639 from asmorkalov:as/android_ocl_rework
#24640 from asmorkalov:as/android_info_lib_cleanup
#24641 from asmorkalov:as/android_samples_drop_mk
#24642 from tomoaki0705:merge_features_aarch64
#24647 from fengyuentau:cuda_sub
#24648 from vrabaud:compilation
#24649 from asmorkalov:as/android_camera2_extact_request
#24651 from asmorkalov:as/android_camera2_resolution_report
#24652 from asmorkalov:as/opencvai
#24653 from savuor:doc_android_tutorials
#24655 from fengyuentau:graph_simplifier_optional_input
#24657 from asmorkalov:as/ffmpeg_timeout_warning
#24658 from smirnov-alexey:as/gapi_ov_get_model_layout
#24659 from MaximSmolskiy:fix-bug-in-ChessBoardDetector-cleanFoundConnectedQuads
#24662 from asmorkalov:as/android_native_camera
#24664 from tomoaki0705:warning_fix_clang
#24665 from opencv-pushbot:gitee/alalek/fix_winpack_vc14
#24666 from zzuliys:4.x
#24667 from MaximSmolskiy:fix-mismatch-and-simplify-code-in-ChessBoardDetector-findQuadNeighbors
#24668 from vrabaud:compilation
#24672 from dkurt:adjust_slice_optional_inputs
#24673 from tomoaki0705:roll_back_23654
#24680 from AleksandrPanov:update_android_mobilenet_tutorial
#24681 from WanliZhong:err_armv8
#24685 from AleksandrPanov:fix_build_grandle
#24689 from FantasqueX:update-libspng-0-7-4
#24692 from savuor:doc_android_tutorial_camera
#24693 from FantasqueX:fix-spng-homepage
#24694 from fengyuentau:matmul_refactor
#24696 from Kumataro:fix24695
#24698 from tomoaki0705:support_arm64_windows
#24700 from savuor:doc_android_win_fix
#24701 from dodo920306:4.x
#24702 from vrabaud:compilation
#24706 from Kumataro:fix24705
#24709 from vpisarev:winograd_mode
#24713 from MaximSmolskiy:improve-icvSmoothHistogram256
#24715 from AleksandrPanov:update_android_opencl_sample
#24716 from asmorkalov:as/spng_warning_fix
#24717 from FantasqueX:zlib-1-3
#24718 from asmorkalov/as/system_spng
#24723 from asmorkalov:as/android_drop_old_toolchain
#24728 from opencv-pushbot:gitee/alalek/samples_python_312
#24730 from mshabunin:fix-mali-opencl
#24732 from asmorkalov:as/android_toolchain_install
#24736 from opencv-pushbot:gitee/alalek/issue_24734
#24737 from opencv-pushbot:gitee/alalek/issue_24735
#24740 from opencv-pushbot:gitee/alalek/ocl_fix_kernel_compilation
#24741 from FantasqueX:fix-zlib-version-parse
#24744 from chacha21:cmake_cuda_help
#24747 from asmorkalov:as/tune_vitb_cuda
#24749 from FantasqueX:fix-openjpeg-version
#24751 from dkurt:d.kurtaev/hotfix_cuda_scale
#24752 from asmorkalov:as/android_aar_with_gradle
#24753 from fengyuentau:einsum_importer
#24754 from FantasqueX:refine-png-version-parsing
#24755 from MaximSmolskiy:fix-typos-in/imgproc/src/contours.cpp
#24759 from jubinchheda:4.x
#24761 from opencv-pushbot:gitee/alalek/test_skip_update_win32
#24763 from opencv-pushbot:gitee/alalek/test_dnn_skip_update_debug
#24764 from asmorkalov:as/python_read_only_docs
#24766 from asmorkalov:update_version_4.9.0-pre
#24767 from opencv-pushbot:gitee/alalek/update_ffmpeg_4.x
#24769 from asmorkalov:as/calib_doc_update
#24771 from FantasqueX:remove-is-clang
#24775 from opencv-pushbot:gitee/alalek/build_python2_disable_by_default
#24776 from opencv-pushbot:gitee/alalek/docs_exclude_bindings_utils
#24778 from asmorkalov:as/win_msmf_stream_type_check

Previous "Merge 4.x": #24486

force_builders=Linux OpenCL,Linux AVX2,Linux32,Win32

asmorkalov and others added 30 commits October 31, 2023 11:23
Backport to 4.x: patchNaNs() SIMD acceleration opencv#24480

backport from opencv#23098
connected PR in extra: [opencv#1118@extra](opencv/opencv_extra#1118)

### This PR contains:
* new SIMD code for `patchNaNs()`
* CPU perf test

<details>
<summary>Performance comparison</summary>

Geometric mean (ms)

|Name of Test|noopt|sse2|avx2|sse2 vs noopt (x-factor)|avx2 vs noopt (x-factor)|
|---|:-:|:-:|:-:|:-:|:-:|
|PatchNaNs::OCL_PatchNaNsFixture::(640x480, 32FC1)|0.019|0.017|0.018|1.11|1.07|
|PatchNaNs::OCL_PatchNaNsFixture::(640x480, 32FC4)|0.037|0.037|0.033|1.00|1.10|
|PatchNaNs::OCL_PatchNaNsFixture::(1280x720, 32FC1)|0.032|0.032|0.033|0.99|0.98|
|PatchNaNs::OCL_PatchNaNsFixture::(1280x720, 32FC4)|0.072|0.072|0.070|1.00|1.03|
|PatchNaNs::OCL_PatchNaNsFixture::(1920x1080, 32FC1)|0.051|0.051|0.050|1.00|1.01|
|PatchNaNs::OCL_PatchNaNsFixture::(1920x1080, 32FC4)|0.137|0.138|0.128|0.99|1.06|
|PatchNaNs::OCL_PatchNaNsFixture::(3840x2160, 32FC1)|0.137|0.128|0.129|1.07|1.06|
|PatchNaNs::OCL_PatchNaNsFixture::(3840x2160, 32FC4)|0.450|0.450|0.448|1.00|1.01|
|PatchNaNs::PatchNaNsFixture::(640x480, 32FC1)|0.149|0.029|0.020|5.13|7.44|
|PatchNaNs::PatchNaNsFixture::(640x480, 32FC2)|0.304|0.058|0.040|5.25|7.65|
|PatchNaNs::PatchNaNsFixture::(640x480, 32FC3)|0.448|0.086|0.059|5.22|7.55|
|PatchNaNs::PatchNaNsFixture::(640x480, 32FC4)|0.601|0.133|0.083|4.51|7.23|
|PatchNaNs::PatchNaNsFixture::(1280x720, 32FC1)|0.451|0.093|0.060|4.83|7.52|
|PatchNaNs::PatchNaNsFixture::(1280x720, 32FC2)|0.892|0.184|0.126|4.85|7.06|
|PatchNaNs::PatchNaNsFixture::(1280x720, 32FC3)|1.345|0.311|0.230|4.32|5.84|
|PatchNaNs::PatchNaNsFixture::(1280x720, 32FC4)|1.831|0.546|0.436|3.35|4.20|
|PatchNaNs::PatchNaNsFixture::(1920x1080, 32FC1)|1.017|0.250|0.160|4.06|6.35|
|PatchNaNs::PatchNaNsFixture::(1920x1080, 32FC2)|2.077|0.646|0.605|3.21|3.43|
|PatchNaNs::PatchNaNsFixture::(1920x1080, 32FC3)|3.134|1.053|0.961|2.97|3.26|
|PatchNaNs::PatchNaNsFixture::(1920x1080, 32FC4)|4.222|1.436|1.288|2.94|3.28|
|PatchNaNs::PatchNaNsFixture::(3840x2160, 32FC1)|4.225|1.401|1.277|3.01|3.31|
|PatchNaNs::PatchNaNsFixture::(3840x2160, 32FC2)|8.310|2.953|2.635|2.81|3.15|
|PatchNaNs::PatchNaNsFixture::(3840x2160, 32FC3)|12.396|4.455|4.252|2.78|2.92|
|PatchNaNs::PatchNaNsFixture::(3840x2160, 32FC4)|17.174|5.831|5.824|2.95|2.95|

</details>

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Optimize the Implementation of RVV Universal Intrinsic.
DNN graph fusion with shared nodes opencv#24463

### Pull Request Readiness Checklist

For now, nodes from matched pattern are removed during the matching process so if nodes are used in similar subgraph, they cannot be found.

required for opencv#24397

**Merge with extra**: opencv/opencv_extra#1115

A part from [model_name ](https://github.com/onnx/models/blob/main/vision/object_detection_segmentation/fcn/model/fcn-resnet101-11.onnx) with two Resize subgraphs with shared nodes:
![image](https://github.com/opencv/opencv/assets/25801568/611d89d9-12fb-4add-9218-13b10d2c086a)

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
…_dnn_sample

Added PyTorch fcnresnet101 segmentation conversion cases opencv#24397

We write a sample code about transforming Pytorch fcnresnet101 to ONNX running on OpenCV.

The input source image was shooted by ourself.

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [X] I agree to contribute to the project under Apache 2 License.
- [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [X] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
* improve and refactor softmax layer

* fix building error

* compatible region layer

* fix axisStep when disable SIMD

* fix dynamic array

* try to fix error

* use nlanes from VTraits

* move axisBias to srcOffset

* fix bug caused by axisBias

* remove macro

* replace #ifdef with #if for CV_SIMD
Get the SSE2 condition match the emmintrin.h inclusion condition. opencv#24495

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
dnn onnx: add instance norm layer opencv#24378

Resolves opencv#24377
Relates opencv#24092 (comment)

| Perf | multi-thread | single-thread |
| - | - | - |
| x: [2, 64, 180, 240] | 3.95ms | 11.12ms |

Todo:

- [x] speed up by multi-threading
- [x] add perf
- [x] add backend: OpenVINO
- [x] add backend: CUDA
- [x] add backend: OpenCL (no fp16)
- [ ] add backend: CANN (will be done via opencv#24462)


### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake

```
force_builders=Linux OpenCL,Win64 OpenCL,Custom
buildworker:Custom=linux-4
build_image:Custom=ubuntu:18.04
modules_filter:Custom=none
disable_ipp:Custom=ON
```
Added scripts for creating an AAR package and a local Maven repository with OpenCV library opencv#24456

Added scripts for creating an AAR package and a local Maven repository with OpenCV library.

The build_java_shared_aar.py script creates AAR with Java + C++ shared libraries.
The build_static_aar.py script creates AAR with static C++ libraries.

The scripts use an Android project template. The project is almost a default Android AAR library project with empty Java code and one empty C++ library. Only build.gradle.template and CMakeLists.txt.template files contain significant changes.

See README.md for more information.
dnn (onnx): add subgraph fusion tests opencv#24500

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
…tractoor_java_ctor

Added Java bindings for BOWImgDescriptorExtractor constructor
Add JavaScript bindings for SimpleBlobDetector
Einsum Layer Performance Test opencv#24445

## This PR adds performance tests for Einsum Layer. See below results of performance test on different inputs

**Notation:**
- WX: windows10_x64
- MX: macos_x64
- MA: macos_arm64
- UX: ubuntu_x64
- UA: ubuntu_arm64

All data in ms (milliseconds).
Gemm is backend for matrix multiplication

---

Benchmarks:


| Equation                | Inputs Mat Dims                   | UX (ms)        | UA (ms) | MX (ms) | MA (ms) | WX (ms) |
|-------------------------|-----------------------------------|----------------|---------|---------|---------|---------|
| "ij, jk -> ik"          | [2, 3], [3,2]                     | 0.04 ± 0.00    | -       | -       | -       | -       |
| "ij, jk -> ik"          | [20, 30], [30,20]                 | 0.08 ± 0.00    | -       | -       | -       | -       |
| "ij, jk -> ik"          | [113, 127], [127,113]             | 2.41 ± 0.05    | -       | -       | -       | -       |
| "imkj, injs -> imnks"   | [1, 4, 7, 9], [1, 5, 9, 8]        | 0.11 ± 0.00    | -       | -       | -       | -       |
| "imkj, injs -> imnks"   | [1, 4, 70, 90], [1, 5, 90, 80]    | 15.49 ± 0.46   | -       | -       | -       | -       |
| "imkj, injs -> imnks"   | [1, 4, 73, 91], [1, 5, 91, 57]    | 11.53 ± 0.06   | -       | -       | -       | -       |
| "ij -> i"               | [30, 40]                          | 0.03 ± 0.00    | -       | -       | -       | -       |
| "ij -> i"               | [113, 374]                        | 0.13 ± 0.00    | -       | -       | -       | -       |
| "...ij -> ...i"         | [30, 40]                          | 0.03 ± 0.00    | -       | -       | -       | -       |
| "...ij -> ...i"         | [113, 374]                        | 0.13 ± 0.00    | -       | -       | -       | -       |
| "...ij, ...jk -> ...ik" | [40, 50], [50,80]                 | 0.37 ± 0.01    | -       | -       | -       | -       |
| "...ij, ...jk -> ...ik" | [47, 51], [51, 83]                | 0.43 ± 0.01    | -       | -       | -       | -       |

-----

### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [ ] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
Fix gstreamer backend with manual pipelines opencv#24243

- Fix broken seeking in audio/video playback
- Fix broken audio playback
- Fix unreliable seeking
- Estimate frame count if it is not available directly
- Return -1 for frame count and fps if it is not available. 
- Return 0 for fps if the video has variable frame rate
- Enable and fix tests


### Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [X] I agree to contribute to the project under Apache 2 License.
- [X] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [X] The PR is proposed to the proper branch
- [-] There is a reference to the original bug report and related work => Reproducible test provided
- [-] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [X] The feature is well documented and sample code can be built with the project CMake

1. Download two test videos:

```bash
wget https://github.com/ietf-wg-cellar/matroska-test-files/raw/master/test_files/test1.mkv
wget https://test-videos.co.uk/vids/jellyfish/mkv/360/Jellyfish_360_10s_5MB.mkv
```

2. I modified a OpenCV videoio sample to demonstrate the problem, here it is the patch: http://dpaste.com//C9MAT2K6W

3. Build the sample, on Ubuntu:

```bash
g++ -g videocapture_audio_combination.cpp -I/usr/include/opencv4 `pkg-config --libs --cflags opencv4` -o videocapture_audio_combination
```

4. Play an audio stream with seeking BEFORE the fix:

```bash
$ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"[ERROR:0@0.009] global cap.cpp:164 open VIDEOIO(GSTREAMER): raised OpenCV exception:

OpenCV(4.8.0-dev) ./modules/videoio/src/cap_gstreamer.cpp:153: error: (-215:Assertion failed) ptr in function 'get'


[ WARN:0@0.009] global cap.cpp:204 open VIDEOIO(GSTREAMER): backend is generally available but can't be used to capture by name
ERROR! Can't to open file: filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink
```

5. Play a video stream with seeking BEFORE the fix:

```bash
$ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1"
[ WARN:0@0.034] global cap_gstreamer.cpp:1728 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300
CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100
CAP_PROP_AUDIO_TOTAL_CHANNELS: 0
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1898 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
[ WARN:0@0.034] global cap_gstreamer.cpp:1817 getProperty OpenCV | GStreamer: CAP_PROP_POS_MSEC property result may be unrealiable: opencv#19025
Timestamp: 0.6218
Timestamp: 33.1085
Timestamp: 67.1274
Timestamp: 100.1182
Timestamp: 133.1204
Timestamp: 167.1195
Timestamp: 200.1161
Timestamp: 233.1147
Timestamp: 267.1194
Timestamp: 300.1202
[ WARN:0@0.338] global cap_gstreamer.cpp:1949 setProperty OpenCV | GStreamer warning: GStreamer: unable to seek
0:00:00.338215907 3892572 0x5592899c7580 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: Internal data stream error.
0:00:00.338235884 3892572 0x5592899c7580 WARN                 basesrc gstbasesrc.c:3127:gst_base_src_loop:<filesrc0> error: streaming stopped, reason not-linked (-1)
0:00:00.338264287 3892572 0x5592899c7580 WARN                   queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: Internal data stream error.
0:00:00.338270329 3892572 0x5592899c7580 WARN                   queue gstqueue.c:992:gst_queue_handle_sink_event:<queue0> error: streaming stopped, reason not-linked (-1)
[ WARN:0@0.339] global cap_gstreamer.cpp:2784 handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module filesrc0 reported: Internal data stream error.
[ WARN:0@0.339] global cap_gstreamer.cpp:1199 startPipeline OpenCV | GStreamer warning: unable to start pipeline
Number of audio samples: 0
Number of video frames: 10
[ WARN:0@0.339] global cap_gstreamer.cpp:1164 isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
```

6. Play an audio stream with seeking AFTER the fix:

```bash
$ ./videocapture_audio_combination --audio "filesrc location=test1.mkv ! queue ! matroskademux name=demux demux.audio_0 ! decodebin ! audioconvert ! appsink"CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 48000
CAP_PROP_AUDIO_TOTAL_CHANNELS: 2
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.025] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
Timestamp: 0.0000
Timestamp: 24.0000
Timestamp: 48.0000
Timestamp: 72.0000
Timestamp: 96.0000
Timestamp: 120.0000
Timestamp: 144.0000
Timestamp: 168.0000
Timestamp: 192.0000
Timestamp: 216.0000
Timestamp: 3500.0000
Timestamp: 3504.0000
Timestamp: 3528.0000
Timestamp: 3552.0000
Timestamp: 3576.0000
Timestamp: 3600.0000
Timestamp: 3624.0000
Timestamp: 3648.0000
Timestamp: 3672.0000
Timestamp: 3696.0000
Timestamp: 3720.0000
Timestamp: 3744.0000
Timestamp: 3768.0000
Timestamp: 3792.0000
Timestamp: 3816.0000
Timestamp: 3840.0000
Timestamp: 3864.0000
Timestamp: 3888.0000
Timestamp: 3912.0000
Timestamp: 3936.0000
```

7. Play a video stream with seeking AFTER the fix:

```bash
$ ./videocapture_audio_combination --audio "filesrc location=Jellyfish_360_10s_5MB.mkv ! queue ! matroskademux name=demux demux.video_0 ! decodebin ! videoconvert ! video/x-raw, format=BGR ! appsink drop=1"
[ WARN:0@0.033] global cap_gstreamer.cpp:1746 open OpenCV | GStreamer warning: Cannot query video position: status=1, value=22, duration=300
CAP_PROP_AUDIO_DATA_DEPTH: CV_16S
CAP_PROP_AUDIO_SAMPLES_PER_SECOND: 44100
CAP_PROP_AUDIO_TOTAL_CHANNELS: 0
CAP_PROP_AUDIO_TOTAL_STREAMS: [ WARN:0@0.034] global cap_gstreamer.cpp:1903 getProperty OpenCV | GStreamer: CAP_PROP_AUDIO_TOTAL_STREAMS property is not supported
0
Timestamp: 0.0000
Timestamp: 33.0000
Timestamp: 67.0000
Timestamp: 100.0000
Timestamp: 133.0000
Timestamp: 167.0000
Timestamp: 200.0000
Timestamp: 233.0000
Timestamp: 267.0000
Timestamp: 300.0000
0:00:00.335931693 3893501 0x55bbe76ad920 WARN      matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.335952823 3893501 0x55bbe76ad920 WARN      matroskareadcommon matroska-read-common.c:759:gst_matroska_read_common_parse_skip:<demux:sink> Unknown CueTrackPositions subelement 0xf0 - ignoring
0:00:00.335988029 3893501 0x55bbe76ad920 WARN                 basesrc gstbasesrc.c:1742:gst_base_src_perform_seek:<filesrc0> duplicate event found 184
Timestamp: 3467.0000
Timestamp: 3500.0000
Timestamp: 3533.0000
Timestamp: 3567.0000
Timestamp: 3600.0000
Timestamp: 3633.0000
Timestamp: 3667.0000
Timestamp: 3700.0000
Timestamp: 3733.0000
Timestamp: 3767.0000
Timestamp: 3800.0000
Timestamp: 3833.0000
Timestamp: 3867.0000
Timestamp: 3900.0000
Timestamp: 3933.0000
Timestamp: 3967.0000
Timestamp: 4000.0000
Timestamp: 4033.0000
Timestamp: 4067.0000
Timestamp: 4100.0000
```
Commutative rules for DNN subgraphs fusion opencv#24483

### Pull Request Readiness Checklist

related: opencv#24463 (comment)

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
      Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
…ls_from_docs

Exclude RVV UI internals from Doxygen documentation.
Link lib_acl_op_compiler when compile with CANN
…ource_ffmpeg

Use video stream fps first in FFmpeg backend for VideoCapture
asmorkalov and others added 2 commits December 27, 2023 19:43
…pe_check

Reworked camera stream check to exclude non-video formats
modules/3d/include/opencv2/3d.hpp Outdated Show resolved Hide resolved
modules/3d/include/opencv2/3d.hpp Outdated Show resolved Hide resolved
modules/3d/include/opencv2/3d.hpp Outdated Show resolved Hide resolved
modules/calib/include/opencv2/calib.hpp Show resolved Hide resolved
doc/pattern_tools/test_charuco_board.py Show resolved Hide resolved
modules/dnn/perf/perf_net.cpp Show resolved Hide resolved
modules/dnn/perf/perf_net.cpp Show resolved Hide resolved
modules/dnn/perf/perf_net.cpp Show resolved Hide resolved
modules/dnn/perf/perf_net.cpp Show resolved Hide resolved
samples/dnn/segmentation.py Outdated Show resolved Hide resolved
@opencv-alalek
Copy link
Contributor

@opencv-alalek
Copy link
Contributor

@asmorkalov
Copy link
Contributor Author

MSVS warnings - they existed before merge. Will fix with the next PR.
Need to tune failed OpenCL tests. The test related to newly added model instead old Torch. Let's fix with dedicated PR.

@asmorkalov asmorkalov merged commit c739117 into opencv:5.x Jan 22, 2024
20 of 25 checks passed
@asmorkalov asmorkalov mentioned this pull request Jan 23, 2024
@dkurt dkurt added this to the 5.0 milestone Apr 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet