QA_in_OpenCV

Alexander Alekhin edited this page Jun 16, 2016 · 3 revisions
Clone this wiki locally

QA in OpenCV

Introduction

This page describes QA and continuous integration in the OpenCV project. OpenCV is a large library and requires different types of testing on different levels. For C++ API testing GoogleTest cross-platform unit-testing framework is used, for Java APIJUnit for Android. For the automation BuildBot continuous integration system is used. We’ll cover some basic usage principles here, but for the detailed information about the mentioned frameworks you should visit respective web-sites.

C++ tests

You can start from the section 9 in OpenCV Coding Style Guide.

The same tests are used on every supported platform: Windows, Linux, MacOS and Android. Majority of testing is performed on C++ level, because it is the major language in OpenCV project. As mentioned above, they are implemented on top of Google’s unit-testing framework, named GoogleTest. There are two major types of C++ tests: accuracy=regression tests and performance tests. C++ tests are compiled into console applications, two for every module: opencv_test_<MODULE_NAME> and opencv_perf_<MODULE_NAME>. These applications may be built for every supported platform (Win/Lin/Mac/Android) and be run in a shell. On Android adb shell is used.

Please note that you should set OPENCV_TEST_DATA_PATH environment variable (see section 9 in OpenCV Coding Style Guilde). On Android you should copy test data to the /sdcard/opencv_testdata directory:

git clone git://github.com/opencv/opencv_extra.git .
adb push ./testdata /sdcard/opencv_testdata

Accuracy Tests

OpenCV library is actively developed, hence a set of regression tests is required. At the same time conformance between different platforms (both software and hardware) should be controlled, this is vital for keeping computer vision algorithms portable. For these two purposes OpenCV contains so-called accuracy tests. They are unit-tests that execute OpenCV functions with different parameters (input/output matrices sizes, number of channels, algorithm parameters and finally input data) and test it against some reference/desired output.

Every OpenCV module (core, imgproc, etc) has a set of accuracy tests, and their sources can be found in OpenCV repository in the test subdirectory of http://github.com/opencv/opencv/tree/master/modules//test, for example http://github.com/opencv/opencv/tree/master/modules/core/test and http://github.com/opencv/opencv/tree/master/modules/imgproc/test.

Performance Tests

Performance tests are in the corresponding perf directories: http://github.com/opencv/opencv/tree/master/modules/core/perf, http://github.com/opencv/opencv/tree/master/modules/imgproc/perf.

  1. How to read/write perf tests: HowToWritePerfTests.
  2. How to execute/analyse perf tests: HowToUsePerfTests.

Java Tests

BuildBot

OpenCV’s continuous integration system is available on www.build.opencv.org. You can start from the Waterfall and Console links in the top-menu. All accuracy, performance and Java tests are executed on a daily (nightly) basis by the BuildBot. There are also separate build-slaves for documentation, distribution package, test coverage calculation and some others. Additionally a small subset of builders is run on every commit to the OpenCV source code repository.

Documentation

Documentation update check list for public releases:

  1. Check all images manually. Update images, that include OpenCV version, if it is needed by context;
  2. Check all *.rst files. Find docs with links to old versions of OpenCV: 2.4.3, 2.4.2, 2.4.1, 2.4.0, 2.3.1, 2.3.0, etc and update if needed;
    Search command for Unix/Linux systems:
    grep -R 2\\.[3-4]\\.[0-2] *