ONNX conformance test results

[rogday]

System information (version)

• OpenCV => 4.5.4
• Operating System / Platform => Linux
• Compiler => g++ 8.4.0

Detailed description

I ran ONNX conformance tests from official repo, and the found the following issues:

• DataLayer

  • Forward fails doe to wrong size access on DataLayer. 1D inputs fail on DataLayer #25070

• Add

  • lack of broadcasting; Reimplementation of Element-wise layers with broadcasting support #21865

• AveragePool

  • ceil_mode is set by checking the existence of pad_mode; fix ceil_mode for Average/MaxPooling #21159
  • count_include_pad not supported
  • auto_pad is deprecated, but we support it, but not making distinction between same_upper and same_lower

• MaxPool

  • ceil_mode is set by checking the existence of pad_mode; fix ceil_mode for Average/MaxPooling #21159
  • auto_pad is deprecated, but we support it, but not making distinction between same_upper and same_lower
  • we seem to not support dilations

• BatchNormalization

  • mean blob is supported only as const

• Clip

  • min and max shouldn't be differentiable, model error?; Fix issue 22015, let Clip layer support 1-3 inputs #22100
  • defaults are wrong; fix Clip, LeakyReLU, LRN, Split defaults #21152

• Concat

  • 2 input 1-d input mats with dims [2] become 2-d mats with dims [1, 2], axis=-1, hence Concat output dims are [2, 2] instead of [4]

• Constant

  • net is empty, technically, not an error?

• ConvTranspose

  • W blob is supported only as const

• Div

  • does not support 1-d inputs (like everything else); Reimplementation of Element-wise layers with broadcasting support #21865

• ELU

  • not handling alpha parameter; Add alpha parameter to ELU layer #21160, Add alpha parameter to ELU (CUDA) #21161

• Flatten

  • does not flatten before axis, if axis=0 outputs [X, 1] instead of [1, X]; fix Flatten layer #21283

• Gemm

  • B blob is supported only as const, ONNX conformance test results #21078 (comment)

• GRU

  • assumes optional inputs are present
  • support some inputs in const-only mode

• InstanceNormalization

  • S blob is supported only as const

• LeakyReLU

  • default alpha should be 0.01; fix Clip, LeakyReLU, LRN, Split defaults #21152

• LogSoftMax/SoftMax

  • default axis should be -1 - if the opset is v11, otherwise it's 1
  • ..._expanded models should be fused; Add Log/Softmax simplification #21162

• LRN

  • default alpha should be 0.0001; fix Clip, LeakyReLU, LRN, Split defaults #21152

• LSTM

  • assumes optional inputs are present
  • support some inputs in const-only mode

• Max/Min/Mul

  • does not support 1-d inputs (like everything else)

• Pow

  • second blob is supported only as const

• PRelu

  • Slope blob is supported only as const

• ReduceMax/ReduceMean

  • doesn't handle 2-d input
  • does not support reduce along some axis(0, if input is 3-d)
  • doesn't support keepdims=true; Fix a few issues in MaxUnpool, ReduceMean/Max/Sum, Pad  #21259

• ReduceSum

  • might be all above +; DNN: Reduce Layer (add dynamic batch and ReduceSum support)  #22199
  • axes arent handled as attributes, should be optional const input; DNN: Reduce Layer (add dynamic batch and ReduceSum support)  #22199

• Resize

  • only first input should be differentiable, model error?

• Split

  • default axis is set to 1 instead of 0; fix Clip, LeakyReLU, LRN, Split defaults #21152

• Sub

  • lack of broadcasting; Reimplementation of Element-wise layers with broadcasting support #21865
  • does not support 1-d inputs (like everything else)

• Sum

  • doesn't support only 1 input(identity?), add Sum of 1 input #21164
  • does not support 1-d inputs (like everything else)

• Transpose

  • does not reverse dimensions in absence of order, Add default order to Transpose #21163

• Upsample

  • scales blob is supported only as const or attribute

• AND

  • fails while parsing graph. Issues is related to handing booleans, Parsing ONNX with AND layer fails. #24979

Steps to reproduce

Clone ONNX repo, clone my branch, fix paths in the code and run opencv_test_dnn.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[fengyuentau]

• support of TopK (in-progress): dnn: add ONNX TopK #23279
• support of NonMaxSuppression (requires support of dynamic shape): DNN: supports NonMaxSuppression operator from ONNX #22473
• support of LayerNormalization: dnn: add layer normalization for vision transformers #23047
• support of Slice with negative steps: dnn: support ONNX Slice with negative steps by adding and using cv::flipND #22898
• support of Tile: dnn: support ONNX Tile #22809
• support of Scatter and ScatterND: DNN: supports Scatter and ScatterND from ONNX #22529
• support of InstanceNormalization: dnn onnx: add instance norm layer #24378
• support of Gelu: Add GELU layer for vision transformers #23219
• support of Mod: dnn onnx: add mod #24765
• refactor of Reduce: dnn: refactor reduce #23613
• refactor of Expand: dnn: expand refactor with cv::broadcast for onnx models #24295
• refactor of Gemm: dnn: add gemm_layer in place of fully_connected_layer for onnx models #23897
• refactor of MatMul: dnn: refactor ONNX MatMul with fastGemm #24694
• support of ORT contrib op Attention: dnn: add attention layer #24476
• support of ORT contrib op QLinearSoftmax: Support ONNX operator QLinearSoftmax in dnn #23655

[asmorkalov]

@fengyuentau I believe "Gemm B blob is supported only as const" is complete?

[fengyuentau]

@fengyuentau I believe "Gemm B blob is supported only as const" is complete?

No, B can be either const or variable. See

opencv/modules/dnn/src/layers/gemm_layer.cpp

Lines 210 to 215 in 76548e2

	if (const_B) {
	CV_CheckGT(packed_B.size(), static_cast<size_t>(0), "DNN/Gemm: constant B is not pre-packed");
	fastGemm(trans_a, M, N, K, alpha, A.ptr<const float>(), na, packed_B.data(), 1.f, Y.ptr<float>(), N, opt);
	} else {
	fastGemmBatch(trans_a, trans_b, alpha, A, inputs[1], 1.f, Y, opt);
	}

[asmorkalov]

Related: #25442