The objective of this project is to demonstrate object localization via template matching using OpenCV-Python built-in functionalities.
Template matching is a technique for finding areas of an image that are similar to the template of an object of interest, such as a person, vehicle, animal, etc. The goal of template matching is to detect and localize the template image (T) in a source image S:
- The source Image (S), which is the image where the template may be localized
- The template Image (T), which is the image that is to be found in the source image.
Template matching is useful when the shape, orientation and scale of the object of interest in the new imaged scene rains almost the same as what is captured in its template image. Most fixed-template matching approach are based on computing a cross-similarity metric between the template and target images and detecting any peaks in the metric values. In particular, template matching apply a sliding window over all the pixels in the source image order to detect the position of cross-similarity metric peak value between a template and template-size patches of target image, at each of its pixels, and locate the best match, if the template image is present in the target image. The absence of such a significant peak indicates that the template image is likely not present in the target image.
The applied template matching process can be outlined as follows:
- The template image simply slides over the input image (as in 2D convolution)
- The template and patch of input image under the template image are compared.
- The result obtained is compared with the threshold
- If the result is greater than threshold, the portion will be marked as detected.
The OpenCV Python API for the template matching functionality is as follows:
result=cv.matchTemplate(image, templ, method[, result[, mask]])
The Parameters are as follows:
- Input:
- image: Image where the search is running.
- template: Searched template. It must be not greater than the source image and have the same data type.
- result: Map of comparison results. If image is W×H and template is w×h then result is (W−w+1)×(H−h+1)
- method: Parameter specifying the comparison method
- The cross-similarity between the template and source image patch can be assessed based in several suitable metrics, which can be grouped as follows:
- Mean squared error with or without normalization
- Cross correlation with or without normalization
- OpenCV offers the six matching metrics, please see TemplateMatchModes for more information.
- The cross-similarity between the template and source image patch can be assessed based in several suitable metrics, which can be grouped as follows:
- mask: Optional mask. It must have the same size as template.
- Output:
- result: Map of comparison results. It must be single-channel 32-bit floating-point. If image is W×H and template is w×h, then result is (W−w+1)×(H−h+1)
The template and source images used to demonstrate the template matching development process are illustrated next.
In this section, we shall outline the steps involved in the developing and illustrating the template matching application using OpenCV with Python API.
-
Author: Mohsen Ghazel (mghazel)
-
Date: March 29th, 2021
-
Project: Object Recognition via Template Matching:
-
The objective of this project is to demonstrate object localization via template matching using OpenCV-Python built-in functionalities:
- Template matching is a pixel-based technique for detecting and recognizing a template image of an object of interest within a newly acquired sources image, that may or may not contain the template image.
- We shall develop the template matching process and illustrate intermediate and final results.
#====================================================== # Python imports and environment setup #====================================================== # opencv import cv2 # numpy import numpy as np # matplotlib import matplotlib.pyplot as plt import matplotlib.image as mpimg # input/output OS import os # Use %matplotlib notebook to get zoom-able & resize-able notebook. # - This is the best for quick tests where you need to work interactively. %matplotlib notebook #====================================================== # Global variales #====================================================== # Threshold for the normalized correlation # coefficient similarity metric used by the template # matching method: # - cv2.TM_CCOEFF_NORMED #------------------------------------------------------ # - values greater thain this threshold are considered # a positive template matching. #------------------------------------------------------ # seta high-confidence threshold CCOEFF_NORMED_THRESHOLD = 0.99 #====================================================== # Test imports and display package versions #====================================================== # Testing the OpenCV version print("OpenCV : ",cv2.__version__) # Testinng the numpy version print("Numpy : ",np.__version__) OpenCV : 3.4.8 Numpy : 1.19.2
Next, we read and visualize the input data, consisting of the template and source images, as discussed above.
#---------------------------------------------------- # Read the template image: #---------------------------------------------------- # template image file name template_img_file_path = "../data/barcelona-team/messi.jpg" # check if the template image file exists if(os.path.exists(template_img_file_path) == 0): print('Template image file name DOES NOT EXIST! = ' + template_img_file_path) # Read the template of the image of the target image template_img = cv2.imread(template_img_file_path, cv2.IMREAD_COLOR) # create a figure and set its axis fig_size = (3, 2) # create the figure plt.figure(figsize=fig_size) # axis off plt.axis('off') # display the template image plt.imshow(cv2.cvtColor(template_img, cv2.COLOR_BGR2RGB)) # set the title plt.title('Template image - Player: Messi', fontsize = 8) # show the image plt.show() # check the shape of the template image template_img_width, template_img_height, template_img_depth = template_img.shape # display a message print("Template image read successfully!") print("- Its size: (height = {0}, width = {2}, depth = {2})".\\ format(template_img_width, template_img_height, template_img_depth)) # convert to grayscale if it is RGB if ( template_img_depth == 3): template_img_gray = cv2.cvtColor(template_img,cv2.COLOR_RGB2GRAY) else: template_img_gray = template_img # display a message print("- Template image converted to grayscale") Template image read successfully! - Its size: (height = 64, width = 3, depth = 3) Template image converted to grayscale
#---------------------------------------------------- # Read the source image: #---------------------------------------------------- # the source image file name reference_img_file_path = "../data/barcelona-team/team.jpg" # check if the reference image file exists if(os.path.exists(reference_img_file_path) == 0): print('Reference image file name DOES NOT EXIST! = ' + reference_img_file_path) # Read the reference image reference_img = cv2.imread(reference_img_file_path, cv2.IMREAD_COLOR) # create a figure and set its axis fig_size = (5, 4) plt.figure(figsize=fig_size) # axis off plt.axis('off') # display the input image plt.imshow(cv2.cvtColor(reference_img, cv2.COLOR_BGR2RGB)) # set the title plt.title('Source image - Barcelona FC', fontsize = 8) # show the image plt.show() # check the shape of the template image reference_img_width, reference_img_height, reference_img_depth = reference_img.shape # display a message print("Reference image read successfully!") print("- Its size: (height = {0}, width = {2}, depth = {2})".\\ format(reference_img_width, reference_img_height, reference_img_depth)) # convert to grayscale if it is RGB if ( reference_img_depth == 3): reference_img_gray = cv2.cvtColor(reference_img,cv2.COLOR_RGB2GRAY) else: reference_img_gray = reference_img # display a message print("Reference image converted to grayscale") Reference image read successfully! - Its size: (height = 750, width = 3, depth = 3) - Reference image converted to grayscale
We are now ready to apply template matching using OpenCV API:
result=cv.matchTemplate(image, templ, method[, result[, mask]])
#---------------------------------------------------- # Apply template matching #---------------------------------------------------- #- first convert the images to uint8 type # - the template image in grayscale template_img_gray = np.uint8(template_img_gray) # - the reference image in gryscale reference_img_gray = np.uint8(reference_img_gray) #---------------------------------------------------- # perform the actual template matching using: #---------------------------------------------------- # - method: cv2.TM_CCOEFF_NORMED # - this is a commonly used method as correlation-coefficient is # found to be an effective cross-similarity metric #---------------------------------------------------- results = cv2.matchTemplate(reference_img_gray,template_img_gray,cv2.TM_CCOEFF_NORMED)
#---------------------------------------------------- # Process the detected templates and overlay the # their bounding-boxes on the source image #---------------------------------------------------- # display a message print('-----------------------------------------------------------------------') print('Template matching resulst with: CORRELATION-COEFFICENT THRESHOLD= {0}'.\\ format(CCOEFF_NORMED_THRESHOLD)) print('-----------------------------------------------------------------------') #---------------------------------------------------- # - First get the location detected with the # similarity metric #---------------------------------------------------- min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(det_results) # get the TLC of the highest-confidence bounding-box top_left = max_loc # get the BRC of the highest-confidence bounding-box bottom_right = (top_left[0] + template_img_height, top_left[1] + template_img_width) # Check if the cross-correlation value is greater than the # specified cross-correlation threshold, then plot it on the image if ( max_val >= CCOEFF_NORMED_THRESHOLD ): # overlay the bounding-box in: GREEN color cv2.rectangle(reference_img,top_left, bottom_right, (0,255,0), 3) # display a message print('BEST Match: Template is detected source image with MAX-CORRELATION-COEFFICENT = {0}'.\\ format(max_val)) #---------------------------------------------------- # - Now check if there are any other template # detection results #---------------------------------------------------- # find all the detection with cross-correlation execcding the # specified cross-correlation threshold, then plot it on the image x_locs, y_locs = np.where( det_results >= CCOEFF_NORMED_THRESHOLD) # the number of detected templates num_detected_templates = x_locs.shape[0] # display a message print("The total number of detected templates = {0}".format(num_detected_templates)) for counter in range(num_detected_templates): # Ignore the best-detection, which is already plotted if ( y_locs[counter] != max_loc[0] and x_locs[counter] != max_loc[1]): # the TLC of the bbox top_left = (y_locs[counter], x_locs[counter]) # the BRC of the bbox bottom_right = (top_left[0] + template_img_height, top_left[1] + template_img_width) # overlay the bbox cv2.rectangle(reference_img,top_left, bottom_right, (255,255,0), 3) # display a message print('OTHER Match: Template is detected source image with MAX-CORRELATION-COEFFICENT = {0}'.\\ format(max_val)) #---------------------------------------------------- # - Visualize the final template detection results #---------------------------------------------------- # create the figure plt.figure("Template Matching Results"); # display the figure plt.imshow(cv2.cvtColor(reference_img, cv2.COLOR_BGR2RGB)); # display the figure plt.show(); # axis off plt.axis('off'); ----------------------------------------------------------------------- Template matching resulst with: CORRELATION-COEFFICENT THRESHOLD= 0.99 ----------------------------------------------------------------------- BEST Match: Template is detected source image with MAX-CORRELATION-COEFFICENT = 0.9956235885620117 The total number of detected templates = 1
The cross-similarity between the target image T and the reference image I can be assessed based in several suitable mtrics:
- Mean squared error with or without normalization
- Cross correlation with or without normalization
OpenCV offers the following 6 template matching modes:
Enumerations
enum cv::TemplateMatchModes {
cv::TM_SQDIFF = 0,
cv::TM_SQDIFF_NORMED = 1,
cv::TM_CCORR = 2,
cv::TM_CCORR_NORMED = 3,
cv::TM_CCOEFF = 4,
cv::TM_CCOEFF_NORMED = 5
}
type of the template matching operation More...
We explored the different similarity metrics used for template matching:
- The normalized cross-correlation heat-map between the template and reference images illustrated in the fogure below.
- Note that higher cross-correlation sub-regions (windows) are associated with the correct location of the template image (window) in the reference image.
| Template | Template Matching Results | |
| Window |  |
|
The limitations of pixel-based object detection approaches are very similar to those of pixel-based image stitching techniques. As such they tend to be computationally expensive as they process all image pixels, they are also sensitive to changes in illumination, scale and orientation. Furthermore, pixel-based template matching is sensitive to occlusion, as the object needs to be fully visible in the scene image to be detected. These techniques are more suitable for restricted environments where imaging conditions, such as image intensity and viewing angles between the template and images containing this template are the same. However, recently pixel-based template-matching techniques, which are less sensitive to variations in orientation, scale, translation, brightness and contrast have been proposed with some reported success.
We have demonstrated that template matching works well for detecting a fixed-template from a course, acquired under similar conditions. However, template matching has several limitations, including:
- Sensitive to changes in illumination, scale and orientation.
- Sensitive to occlusion, as the object needs to be fully visible in the scene image to be detected.
- Pattern occurrences have to preserve the orientation of the reference pattern image. As a result, it does not work for rotated or scaled versions of the template as a change in shape/size/shear etc. of object with respect to template will give a false match.
- The method is inefficient when calculating the pattern correlation image for medium to large images as the process is time consuming. (* Template matching techniques are more suitable for restricted environments where imaging conditions, such as image intensity and viewing angles between the template and images containing this template are the same.
- However, recently template-matching techniques, which are less sensitive to variations in scale, translation, brightness and contrast have been proposed with some reported success.
- For example, multi-scale template matching is able of detect templates at different scales. That is if the size of the source image patch corresponding to the template image has different size and scale than the used template image.
We propose to explore the following issues related to template matching:
- To study and implement multi-scale template matching is able of detect templates at different scales. That is if the size of the source image patch corresponding to the template image has different size and scale than the used template image
- To study and explore deformable template matching, where the appearance of the object of interest changes in the sources images over time, but we only have a single template:
- For instance, suppose we have a single image, cropped from one of the frames, of a person walking in a video and we would like to detect this person in different frames.
- Clearly, the person appearance, in terms of shape and size and scale, is expected to vary from frame to frame.
- We cannot reliably detect the person in rest of frames from the single template obtained from one frame.
- We shall explore ways to update the template image after each time it is detected from a frame.
- This problem is better solved using moving object tracking but it will be interesting to explore it using adaptive template matching.
- OpenCV. Template Matching. Retrieved from: https://docs.opencv.org/master/d4/dc6/tutorial_py_template_matching.html
- OpenCV. Template Matching. Retrieved from: https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_template_matching/py_template_matching.html
- OpenCV. Template Matching. Retrieved from: https://www.ccoderun.ca/programming/doxygen/opencv_3.2.0/tutorial_template_matching.html
- Adrian Rosebrock. Multi-scale Template Matching using Python and OpenCV. Retrieved from: https://www.pyimagesearch.com/2015/01/26/multi-scale-template-matching-using-python-opencv/
- Ravindu Senaratne. Object Detection on Python Using Template Matching. Retrieved from: https://towardsdatascience.com/object-detection-on-python-using-template-matching-ab4243a0ca62
- SentDex. Template Matching OpenCV Python Tutorial. Retrieved from: https://pythonprogramming.net/template-matching-python-opencv-tutorial/
- Strahinja Zivkovic. Template matching using OpenCV in Python. Retrieved from: http://datahacker.rs/014-template-matching-using-opencv-in-python/
- GeeksforGeeks. Template matching using OpenCV in Python. Retrieved from: https://www.geeksforgeeks.org/template-matching-using-opencv-in-python/