DNN module tests fail due ot missing files, how run DNN module tests?

[diablodale]

I am unable to run the accuracy tests for the DNN module. Many other modules run ok, but DNN fails because it can't find many files. For example

[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v2_coco_2018_03_29.pb

My guess is there is some undocumented step to find these files. They are not in the opencv_extra repo. For example, a file named dnn/tensorflow_inception_graph.pb that is needed does not exist in the repo. The answer is probably simple, I just can't find it in the wiki, forums, readme, etc.

When the answer is clear, I recommend a README be put into the main DNN folder in the extras repo, and in the wiki here on github.

System information (version)

• OpenCV => 4.5.3
• Operating System / Platform => Microsoft Windows [Version 10.0.19043.1165]
• Compiler => MSVC 2019 v16.11.0

Detailed description

1. git clone opencv
2. git checkout 4.5.3
3. git clone opencv_extra
4. git checkout matching 4.5.3
5. config and compile opencv with DNN and building tests
6. Set both OPENCV_DNN_TEST_DATA_PATH and OPENCV_TEST_DATA_PATH are set to my local clone of the extra repo
7. Run the test opencv_test_dnn.exe --gtest_filter=*Tensor*

Result

Note: Google Test filter = *Tensor*
[==========] Running 320 tests from 5 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from Test_TensorFlow
[ RUN      ] Test_TensorFlow.read_inception
[     SKIP ] OpenCV tests: Can't find data file: dnn/tensorflow_inception_graph.pb
[       OK ] Test_TensorFlow.read_inception (1 ms)
[ RUN      ] Test_TensorFlow.inception_accuracy
[     SKIP ] OpenCV tests: Can't find data file: dnn/tensorflow_inception_graph.pb
[       OK ] Test_TensorFlow.inception_accuracy (0 ms)
[ RUN      ] Test_TensorFlow.two_inputs
[       OK ] Test_TensorFlow.two_inputs (1 ms)
[----------] 3 tests from Test_TensorFlow (4 ms total)

[----------] 8 tests from DNNTestNetwork
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow/0, where GetParam() = OCV/OCL
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow/0 (8 ms)
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow/1, where GetParam() = OCV/OCL_FP16
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow/1 (8 ms)
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow_Different_Width_Height/0, where GetParam() = OCV/OCL
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow_Different_Width_Height/0 (8 ms)
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow_Different_Width_Height/1, where GetParam() = OCV/OCL_FP16       
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v1_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v1_TensorFlow_Different_Width_Height/1 (8 ms)
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v2_TensorFlow/0, where GetParam() = OCV/OCL
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v2_coco_2018_03_29.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v2_TensorFlow/0 (9 ms)
[ RUN      ] DNNTestNetwork.MobileNet_SSD_v2_TensorFlow/1, where GetParam() = OCV/OCL_FP16
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_mobilenet_v2_coco_2018_03_29.pb
[       OK ] DNNTestNetwork.MobileNet_SSD_v2_TensorFlow/1 (8 ms)
[ RUN      ] DNNTestNetwork.Inception_v2_SSD_TensorFlow/0, where GetParam() = OCV/OCL
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_inception_v2_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.Inception_v2_SSD_TensorFlow/0 (8 ms)
[ RUN      ] DNNTestNetwork.Inception_v2_SSD_TensorFlow/1, where GetParam() = OCV/OCL_FP16
[     SKIP ] OpenCV tests: Can't find data file: dnn/ssd_inception_v2_coco_2017_11_17.pb
[       OK ] DNNTestNetwork.Inception_v2_SSD_TensorFlow/1 (8 ms)
[----------] 8 tests from DNNTestNetwork (74 ms total)

[----------] 276 tests from Test_TensorFlow_layers
[ RUN      ] Test_TensorFlow_layers.reduce_mean/0, where GetParam() = OCV/OCL
[       OK ] Test_TensorFlow_layers.reduce_mean/0 (10 ms)
[ RUN      ] Test_TensorFlow_layers.reduce_mean/1, where GetParam() = OCV/OCL_FP16
[       OK ] Test_TensorFlow_layers.reduce_mean/1 (2 ms)
...
[ RUN      ] Test_TensorFlow_nets.EAST_text_detection/0, where GetParam() = OCV/OCL
[     SKIP ] OpenCV tests: Can't find data file: dnn/frozen_east_text_detection.pb
[       OK ] Test_TensorFlow_nets.EAST_text_detection/0 (1 ms)
[ RUN      ] Test_TensorFlow_nets.EAST_text_detection/1, where GetParam() = OCV/OCL_FP16
[     SKIP ] OpenCV tests: Can't find data file: dnn/frozen_east_text_detection.pb
[       OK ] Test_TensorFlow_nets.EAST_text_detection/1 (1 ms)
[ RUN      ] Test_TensorFlow_nets.EAST_text_detection/2, where GetParam() = OCV/CPU
[     SKIP ] OpenCV tests: Can't find data file: dnn/frozen_east_text_detection.pb

Expected

No skips of missing files. OK test passed for all results.

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  forum.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there

[jebastin-nadar]

You need to download the models using download_models.py

[diablodale]

Ah, thank you. I'll create a PR to update the docs so it can be more easily found.

[diablodale]

Unfortunately, 15 of the test models are not available. This has been such for 5 days.

Following models have not been downloaded:
* ResNet-50-model.caffemodel
* colorization_deploy_v2.prototxt
* tiny_yolov2.tar.gz
* onnx/models/tiny_yolo2.onnx
* onnx/data/input_tiny_yolo2.pb
* onnx/data/output_tiny_yolo2.pb
* mnist.tar.gz
* onnx/models/cnn_mnist.onnx
* onnx/data/input_cnn_mnist.pb
* onnx/data/output_cnn_mnist.pb
* emotion_ferplus.tar.gz
* onnx/models/emotion_ferplus.onnx
* onnx/data/input_emotion_ferplus.pb
* onnx/data/output_emotion_ferplus.pb
* yolov4-tiny.weights

When I try to download ResNet50 directly with my browser I get...

This item might not exist or is no longer available
This item might have been deleted, expired, or you might not have permission to view it. Contact the owner of this item for more information.

Do we have alternate download locations for these 15? Or remove them and their dependent test cases?

[alalek]

This is expected case that external public links may become broken after years. We don't control external resources.
This is why such tests are optional (they don't fail by default, just skip the test case).
No plans to remove these tests.

[diablodale]

that's not my experience on some of them

[ RUN      ] DNNTestNetwork.ResNet_50/0, where GetParam() = OCV/OCL
Exception thrown at 0x00007FFE5BC74ED9 in opencv_test_dnn.exe: Microsoft C++ exception: cv::Exception at memory location 0x0000001E7DD7E0C0.
unknown file: error: C++ exception with description "OpenCV(3.4.15-dev) ..\modules\dnn\src\caffe\caffe_io.cpp:1176: error: (-2:Unspecified error) FAILED: ReadProtoFromBinaryFile(param_file, param). Failed to parse NetParameter file: C:/repos-nobackup/opencv_extra/testdata/dnn/ResNet-50-model.caffemodel in function 'cv::dnn::ReadNetParamsFromBinaryFileOrDie'
" thrown in the test body.unknown file: error: C++ exception with description "OpenCV(3.4.15-dev) ..\modules\dnn\src\caffe\caffe_io.cpp:1176: error: (-2:Unspecified error) FAILED: ReadProtoFromBinaryFile(param_file, param). Failed to parse NetParameter file: C:/repos-nobackup/opencv_extra/testdata/dnn/ResNet-50-model.caffemodel in function 'cv::dnn::ReadNetParamsFromBinaryFileOrDie'
" thrown in the test body.

[  FAILED  ] DNNTestNetwork.ResNet_50/0, where GetParam() = OCV/OCL (4 ms)

This is caused because the file is no longer available on the internet. The downloader saves the data that is returned (is is html and javascript). Then the DNN module tries to load this .caffemodel file which is actually the html/javascript from the failed download.

The downloader is saving invalid data. And the DNN module is throwing exceptions on the invalid data. Therefore failing tests.

[alalek]

It makes sense to improve downloader to remove/rename broken files if any (which fails hash sum validation).