ERROR input shape not found in function 'cv::dnn::dnn4_v20230620::ONNXImporter::parseEinsum'

[LaurentBerger]

System Information

General configuration for OpenCV 4.8.0-dev =====================================
  Version control:               4.8.0-507-g500fd453a1

  Extra modules:
    Location (extra):            C:/lib/opencv_contrib/modules
    Version control (extra):     4.8.1-49-g0bcbc73b

  Platform:
    Timestamp:                   2023-09-09T13:26:40Z
    Host:                        Windows 10.0.22621 AMD64
    CMake:                       3.26.1
    CMake generator:             Visual Studio 17 2022
    CMake build tool:            C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe
    MSVC:                        1935
    Configuration:               Debug Release

  CPU/HW features:
    Baseline:                    SSE SSE2 SSE3
      requested:                 SSE3
    Dispatched code generation:  SSE4_1 SSE4_2 FP16 AVX AVX2 AVX512_SKX
      requested:                 SSE4_1 SSE4_2 AVX FP16 AVX2 AVX512_SKX
      SSE4_1 (18 files):         + SSSE3 SSE4_1
      SSE4_2 (2 files):          + SSSE3 SSE4_1 POPCNT SSE4_2
      FP16 (1 files):            + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 AVX
      AVX (9 files):             + SSSE3 SSE4_1 POPCNT SSE4_2 AVX
      AVX2 (38 files):           + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2
      AVX512_SKX (8 files):      + SSSE3 SSE4_1 POPCNT SSE4_2 FP16 FMA3 AVX AVX2 AVX_512F AVX512_COMMON AVX512_SKX

  C/C++:
    Built as dynamic libs?:      YES
    C++ standard:                11
    C++ Compiler:                C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.35.32215/bin/Hostx64/x64/cl.exe  (ver 19.35.32215.0)
    C++ flags (Release):         /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /wd4819 /MP  /MD /O2 /Ob2 /DNDEBUG
    C++ flags (Debug):           /DWIN32 /D_WINDOWS /W4 /GR  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /EHa /wd4127 /wd4251 /wd4324 /wd4275 /wd4512 /wd4589 /wd4819 /MP  /MDd /Zi /Ob0 /Od /RTC1
    C Compiler:                  C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.35.32215/bin/Hostx64/x64/cl.exe
    C flags (Release):           /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP   /MD /O2 /Ob2 /DNDEBUG
    C flags (Debug):             /DWIN32 /D_WINDOWS /W3  /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _SCL_SECURE_NO_WARNINGS /Gy /bigobj /Oi  /fp:precise     /MP /MDd /Zi /Ob0 /Od /RTC1
    Linker flags (Release):      /machine:x64  /INCREMENTAL:NO
    Linker flags (Debug):        /machine:x64  /debug /INCREMENTAL
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          cudart_static.lib nppc.lib nppial.lib nppicc.lib nppidei.lib nppif.lib nppig.lib nppim.lib nppist.lib nppisu.lib nppitc.lib npps.lib cublas.lib cudnn.lib cufft.lib -LIBPATH:C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib/x64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 alphamat aruco bgsegm bioinspired calib3d ccalib core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev datasets dnn dnn_objdetect dnn_superres dpm face features2d flann fuzzy gapi hfs highgui img_hash imgcodecs imgproc intensity_transform java line_descriptor mcc ml objdetect optflow phase_unwrapping photo plot python3 quality rapid reg rgbd saliency sfm shape stereo stitching structured_light superres surface_matching text tracking ts video videoio videostab viz wechat_qrcode xfeatures2d ximgproc xobjdetect xphoto
    Disabled:                    world
    Disabled by dependency:      -
    Unavailable:                 cannops cvv freetype hdf julia matlab ovis python2
    Applications:                tests perf_tests examples apps
    Documentation:               doxygen python javadoc
    Non-free algorithms:         YES

  Windows RT support:            NO

  GUI:                           WIN32UI
    Win32 UI:                    YES
    OpenGL support:              YES (opengl32 glu32)
    VTK support:                 YES (ver 9.2.5)

  Media I/O:
    ZLib:                        optimized C:/install/zlib/lib/zlib.lib debug C:/install/zlib/lib/zlibd.lib (ver 1.2.13)
    JPEG:                        build-libjpeg-turbo (ver 2.1.3-62)
      SIMD Support Request:      YES
      SIMD Support:              NO
    WEBP:                        build (ver encoder: 0x020f)
    PNG:                         optimized C:/install/libpng/lib/libpng16.lib debug C:/install/libpng/lib/libpng16d.lib (ver 1.6.40)
    TIFF:                        build (ver 42 - 4.2.0)
    JPEG 2000:                   build (ver 2.5.0)
    OpenEXR:                     build (ver 2.3.0)
    HDR:                         YES
    SUNRASTER:                   YES
    PXM:                         YES
    PFM:                         YES

  Video I/O:
    DC1394:                      NO
    FFMPEG:                      YES (prebuilt binaries)
      avcodec:                   YES (58.134.100)
      avformat:                  YES (58.76.100)
      avutil:                    YES (56.70.100)
      swscale:                   YES (5.9.100)
      avresample:                YES (4.0.0)
    GStreamer:                   NO
    DirectShow:                  YES
    Media Foundation:            YES
      DXVA:                      YES

  Parallel framework:            TBB (ver 2020.3 interface 11103)

  Other third-party libraries:
    Intel IPP:                   2021.11.0 [2021.11.0]
           at:                   C:/lib/build/opencv/3rdparty/ippicv/ippicv_win/icv
    Intel IPP IW:                sources (2021.11.0)
    Lapack:                      NO
    OpenVINO:                    YES (2022.3.0)
    Default DNN backend:         DNN_BACKEND_OPENCV
    Eigen:                       YES (ver ..)
    Custom HAL:                  NO
    Protobuf:                    build (3.19.1)
    Flatbuffers:                 builtin/3rdparty (23.5.9)

  NVIDIA CUDA:                   YES (ver 12.1, CUFFT CUBLAS)
    NVIDIA GPU arch:             86
    NVIDIA PTX archs:

  cuDNN:                         YES (ver 8.8.0)

  OpenCL:                        YES (NVD3D11)
    Include path:                C:/lib/opencv/3rdparty/include/opencl/1.2
    Link libraries:              Dynamic load

  Python 3:
    Interpreter:                 C:/Program Files/Python310/python.exe (ver 3.10.10)
    Libraries:                   optimized C:/Program Files/Python310/libs/python310.lib debug C:/Program Files/Python310/libs/python310_d.lib (ver 3.10.10)
    numpy:                       C:/Users/laurent/AppData/Roaming/Python/Python310/site-packages/numpy/core/include (ver 1.23.5)
    install path:                C:/Users/laurent/AppData/Roaming/Python/Python310/site-packages/cv2/python-3.10

  Python (for build):            C:/Program Files/Python310/python.exe

  Java:
    ant:                         C:/apache-ant-1.10.13/bin/ant.bat (ver 1.10.13)
    Java:                        NO
    JNI:                         C:/Program Files/Java/jdk-17/include C:/Program Files/Java/jdk-17/include/win32 C:/Program Files/Java/jdk-17/include
    Java wrappers:               YES (ANT)
    Java tests:                  YES

  Install to:                    C:/install/opencv
-----------------------------------------------------------------

### Detailed description

try to download onnx model

[ERROR:0@222.418] global onnx_importer.cpp:1033 cv::dnn::dnn4_v20230620::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 2 inputs and 1 outputs: [Einsum]:(onnx_node!/blocks.0/attn/Einsum) from domain='ai.onnx'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
cv2.error: OpenCV(4.8.0-dev) C:\lib\opencv\modules\dnn\src\onnx\onnx_importer.cpp:1052: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20230620::ONNXImporter::handleNode'
> Node [Einsum@ai.onnx]:(onnx_node!/blocks.0/attn/Einsum) parse error: OpenCV(4.8.0-dev) C:\lib\opencv\modules\dnn\src\onnx\onnx_importer.cpp:3251: error: (-215:Assertion failed) ERROR input shape not found in function 'cv::dnn::dnn4_v20230620::ONNXImporter::parseEinsum'

Steps to reproduce

import cv2 as cv

encoder_name = "vit_l_encoder.onnx_sim.onnx"
net_encoder = cv.dnn.readNet(encoder_name)

model
bhwc, hkc - bhwk.zip

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues, forum.opencv.org, Stack Overflow, etc and have not found any solution
• I updated to the latest OpenCV version and the issue is still there
• There is reproducer code and related data files (videos, images, onnx, etc)

[asmorkalov]

Looks like it's not the only issue with the provided model. model diagnostics tool is attached.
vit_l_encoder.onnx_sim.onnx.log

[fengyuentau]

Let me see whether #24694 can solve the issue.

[LaurentBerger]

einsumLayer don't get initializer

with this may be it can help :

void ONNXImporter::parseEinsum(LayerParams& layerParams, const opencv_onnx::NodeProto& node_proto)
{
    std::vector<MatShape> einsumInpShapes;
    for (int j = 0; j < node_proto.input_size(); j++)
    {
        const auto& inputLayerName = node_proto.input(j);
        auto it = outShapes.find(inputLayerName);
        if (it != outShapes.end())
        {
            einsumInpShapes.emplace_back(it->second);
        }
        else
        {
            if (constBlobs.find(node_proto.input(j)) != constBlobs.end())
            {
                Mat priors = getBlob(node_proto, j);

                LayerParams constParams;
                constParams.name = inputLayerName;
                constParams.type = "Const";
                constParams.blobs.push_back(priors);
                opencv_onnx::NodeProto node_proto_ = node_proto;
                opencv_onnx::NodeProto priorsProto;
                priorsProto.add_output(constParams.name);
                addLayer(constParams, priorsProto);

                node_proto_.set_input(j, constParams.name);
                auto it = constBlobs.find(inputLayerName);
                std::vector<int> szB;;
                for (int dimBlob = 0; dimBlob < priors.dims; dimBlob++)
                    szB.push_back(priors.size[dimBlob]);
                if (it != constBlobs.end())
                    einsumInpShapes.emplace_back(szB);
            }
            else
            {
                CV_Error(Error::StsAssert, "ERROR input shape not found");
            }
        }
    }

    CV_CheckFalse(einsumInpShapes.empty(), "ERROR no inputs shapes");
    for (int i = 0; i < einsumInpShapes.size(); i++) {
        layerParams.set("inputShapes" + cv::format("%d", i), DictValue::arrayInt(einsumInpShapes[i].begin(), einsumInpShapes[i].size()));
    }

    // Check if of eqution is valid
    std::string equation = layerParams.get<std::string>("equation");
    CV_CheckFalse(equation.empty(), "Equation is empty");

    // Save number of inputs. We need it in layer initialization
    layerParams.set("inputSize", node_proto.input_size());

    // Save number of outputs. We need it in layer initialization
    layerParams.set("outputSize", node_proto.output_size());

    addLayer(layerParams, node_proto);
}

[asmorkalov]

@fengyuentau Could you handle the initializer issue?

[fengyuentau]

@fengyuentau Could you handle the initializer issue?

Let me have a try.

[fengyuentau]

@LaurentBerger You are right. parseEinsum does not get all constant input shapes. Your code could be a workaround to solve this problem. Could you form your code into a good shape and submit a PR to fix the problem?

---

I think getting these input shapes in getMemoryShapes and forward is bettern than getting from importer, because the inference shape is actually confirm at these stages.