Let me see

RAPP vision module

Most of the robots, that are designed to interact with people, are equipped with one or more video cameras. Interfacing with this sensors is done via rapp::robot::vision class and some API objects, e.g. rapp::object::picture.

Color recognition

Simple, yes useful, application using vision module could be color recognition app. Person puts an object in front of a robot and, after image acquisition, robot recognizes dominant color and tells its name. We will be using OpenCV library to accomplish the task.

The algorithm is rather short and easy to understand:

• capture image
• convert it to cv::Mat representation
• convert image to HSV color space
• cut middle part to further analysis
• calculate number of pixels for each of the color ranges (see also hue scale diagram below)
  • H <= 18 - Red
  • H <= 40 - Orange
  • H <= 69 - Yellow
  • H <= 168 - Green
  • H <= 251 - Blue
  • H <= 318 - Violet
  • H <= 360 - Red
• tell the name of the color

Coding time!

After [creating a new application](Create your first RApp), you have to include few headers. We need rapp::robot::vision module along with OpenCV headers, as well as rapp::robot::communication for user interaction. In header section add:

#include <rapp-robots-api/communication/communication.hpp>
#include <rapp-robots-api/vision/vision.hpp>

#include <opencv2/opencv.hpp>

// convenience alias for camera resolution enum
using cr = rapp::robot::vision::camera_resolution;

and after info object add new one:

int main(int argc, char * argv[]) {
    // create info module
    rapp::robot::info info(argc, argv);
    // create communication module
    rapp::robot::communication comm(argc, argv);
    // create vision module
    rapp::robot::vision vis(argc, argv);

To get image, use capture_image method. As an argument id of the camera should be passed, along with selected resolution and image format. In our case it's camera nr 0, VGA resolution, and we want the image to be encoded as png. We can also inform user, that image is going to be acquired.

    comm.text_to_speech("Show me something!");
    std::this_thread::sleep_for(std::chrono::seconds(2));
    
    rapp::object::picture::Ptr picture;
    picture = vis.capture_image(0, cr::vga, "png");

    comm.text_to_speech("Ok");

Rapp format for pictures is used as an interchange format with other RAPP services, but to process image using OpenCV library it is required to convert it to cv::Mat. It can be done using a combination of picture::byteArray() method with imdecode from OpenCV:

    // convert captured image to OpenCV format
    cv::Mat img;
    img = cv::imdecode(picture->bytearray(), CV_LOAD_IMAGE_ANYDEPTH);

At this point we can work with OpenCV for all further processing. First we have to convert image to HSV color space and cut middle part of the image:

    // change color space to HSV
    cv::cvtColor(img, img, CV_BGR2HSV);

    // check image size and cut middle part
    cv::Size s = img.size();
    int sx = s.width * 0.3;
    int sy = s.height * 0.3;
    int w = 0.4 * s.width;
    int h = 0.4 * s.height;

    img = img(cv::Rect(sx, sy, w, h));

Next part is to calculate histogram for hue channel. We set the ranges to ones defined at the beginning, but we have to remember, that in OpenCV hue channel is scaled to half of the values, from 0-360 to 0-180. All the values for cv::calcHist must be carefully filled. For further information about histogram calculation, refer to the OpenCV docs.

    // create ranges for histogram; note, that hue values are divided by 2 in OpenCV
    int hbins = 7; 
    int histSize[] = { hbins }; // set number of histogram bins
    float hranges[] = {0, 10, 29, 35, 85, 126, 160, 181};
    const float * ranges[] = { hranges };
    int channels[] = { 0 }; // use only H channel
    int numImages = 1; // we have only one image
    bool uniform = false; // non-uniform bin ranges
    bool accumulate = false; // do not accumulate output values
    int dims = 1; // we compute histogram for only one channel

    // calculate histogram for hue channel
    cv::MatND hist;
    cv::calcHist(&img, numImages, channels, cv::Mat(), hist, dims, histSize, ranges, uniform, accumulate);

As a last step we have to add first and last bin of the histogram (as both are for the red color) and find maximum value. After that robot is ready to tel the color of the object.

    // add first and last bin, as both are for red color
    hist.at<float>(0) += hist.at<float>(hbins-1);

    // find maximum value
    float max = -1;
    int maxid = -1;
    for (int i = 0; i < hbins - 2; ++i) {
        if (hist.at<float>(i) > max) {
            max = hist.at<float>(i);
            maxid = i;
        }
    }

    // tell the name of the dominant color
    std::vector<std::string> colors = {"red", "orange", "yelow", "green", "blue", "violet", "red"};
    comm.text_to_speech(std:string("I can see the ") + colors[maxid] + " color!");

    return 0;
}

CMake changes

In order to use external libraries, two things must be changed in CMakeLists.txt file. First, you have to find the OpenCV library:

find_package(OpenCV REQUIRED)

Proper libraries must be also linked to our executable, using create_rapp macro:

create_rapp(
    NAME ${PROJECT_NAME}
    SRCS main.cpp
    LIBS ${OpenCV_LIBS}
)

Now whole project should build without problems. For full source code, go to let_me_see sample app.