Skip to content
Permalink
Browse files

Allow use of masks for unit test control images

Masks set which pixels in the control image should be tested and
an optional tolerance for each pixel. This is done via the colors
in the mask image - white pixels are ignored, black must be an
exact match, and gray levels represent the maximum color component
deviation for that pixel.

This should replace the fragile anomaly images, in that a single
control image with a suitable mask will not be susceptible to
antialiasing differences, etc.

A new script (scripts/generate_test_mask_image.py) is included which
either creates a new mask or modifies an existing mask to handle
an acceptable rendered image.

Ultimately, masking along with multi render checks for specific
platform differences should be flexible enough to meet our needs.
  • Loading branch information
nyalldawson committed Feb 18, 2015
1 parent bf56457 commit 854c0b8bab4499f8f02795cbc140f6ccb5d50793
Showing with 123 additions and 11 deletions.
  1. +86 −0 scripts/generate_test_mask_image.py
  2. +37 −11 src/core/qgsrenderchecker.cpp
  3. BIN tests/testdata/control_images/expected_atlas_autoscale1/{precise → }/expected_atlas_autoscale1.png
  4. BIN tests/testdata/control_images/expected_atlas_autoscale1/expected_atlas_autoscale1_mask.png
  5. BIN tests/testdata/control_images/expected_atlas_autoscale1/quantal/expected_atlas_autoscale1.png
  6. BIN tests/testdata/control_images/expected_atlas_autoscale2/{precise → }/expected_atlas_autoscale2.png
  7. BIN tests/testdata/control_images/expected_atlas_autoscale2/expected_atlas_autoscale2_mask.png
  8. BIN tests/testdata/control_images/expected_atlas_autoscale2/quantal/expected_atlas_autoscale2.png
  9. BIN ...ntrol_images/expected_atlas_autoscale_old_api1/{precise → }/expected_atlas_autoscale_old_api1.png
  10. BIN ...tdata/control_images/expected_atlas_autoscale_old_api1/expected_atlas_autoscale_old_api1_mask.png
  11. BIN ...ta/control_images/expected_atlas_autoscale_old_api1/quantal/expected_atlas_autoscale_old_api1.png
  12. BIN ...ntrol_images/expected_atlas_autoscale_old_api2/{precise → }/expected_atlas_autoscale_old_api2.png
  13. BIN ...tdata/control_images/expected_atlas_autoscale_old_api2/expected_atlas_autoscale_old_api2_mask.png
  14. BIN ...ta/control_images/expected_atlas_autoscale_old_api2/quantal/expected_atlas_autoscale_old_api2.png
  15. BIN tests/testdata/control_images/expected_atlas_filtering1/{precise → }/expected_atlas_filtering1.png
  16. BIN tests/testdata/control_images/expected_atlas_filtering1/expected_atlas_filtering1_mask.png
  17. BIN tests/testdata/control_images/expected_atlas_filtering1/quantal/expected_atlas_filtering1.png
  18. BIN tests/testdata/control_images/expected_atlas_fixedscale1/{precise → }/expected_atlas_fixedscale1.png
  19. BIN tests/testdata/control_images/expected_atlas_fixedscale1/expected_atlas_fixedscale1_mask.png
  20. BIN tests/testdata/control_images/expected_atlas_fixedscale1/quantal/expected_atlas_fixedscale1.png
  21. BIN tests/testdata/control_images/expected_atlas_fixedscale2/{precise → }/expected_atlas_fixedscale2.png
  22. BIN tests/testdata/control_images/expected_atlas_fixedscale2/expected_atlas_fixedscale2_mask.png
  23. BIN tests/testdata/control_images/expected_atlas_fixedscale2/quantal/expected_atlas_fixedscale2.png
  24. BIN ...rol_images/expected_atlas_fixedscale_old_api1/{precise → }/expected_atlas_fixedscale_old_api1.png
  25. BIN ...ata/control_images/expected_atlas_fixedscale_old_api1/expected_atlas_fixedscale_old_api1_mask.png
  26. BIN .../control_images/expected_atlas_fixedscale_old_api1/quantal/expected_atlas_fixedscale_old_api1.png
  27. BIN ...rol_images/expected_atlas_fixedscale_old_api2/{precise → }/expected_atlas_fixedscale_old_api2.png
  28. BIN ...ata/control_images/expected_atlas_fixedscale_old_api2/expected_atlas_fixedscale_old_api2_mask.png
  29. BIN .../control_images/expected_atlas_fixedscale_old_api2/quantal/expected_atlas_fixedscale_old_api2.png
  30. BIN tests/testdata/control_images/expected_atlas_hiding1/expected_atlas_hiding1_mask.png
  31. BIN tests/testdata/control_images/expected_atlas_hiding2/expected_atlas_hiding2_mask.png
  32. BIN ...control_images/expected_atlas_predefinedscales1/{precise → }/expected_atlas_predefinedscales1.png
  33. BIN ...estdata/control_images/expected_atlas_predefinedscales1/expected_atlas_predefinedscales1_mask.png
  34. BIN ...data/control_images/expected_atlas_predefinedscales1/quantal/expected_atlas_predefinedscales1.png
  35. BIN ...control_images/expected_atlas_predefinedscales2/{precise → }/expected_atlas_predefinedscales2.png
  36. BIN ...estdata/control_images/expected_atlas_predefinedscales2/expected_atlas_predefinedscales2_mask.png
  37. BIN ...data/control_images/expected_atlas_predefinedscales2/quantal/expected_atlas_predefinedscales2.png
  38. BIN tests/testdata/control_images/expected_atlas_sorting1/{precise → }/expected_atlas_sorting1.png
  39. BIN tests/testdata/control_images/expected_atlas_sorting1/expected_atlas_sorting1_mask.png
  40. BIN tests/testdata/control_images/expected_atlas_sorting1/quantal/expected_atlas_sorting1.png
  41. BIN tests/testdata/control_images/expected_atlas_sorting2/{precise → }/expected_atlas_sorting2.png
  42. BIN tests/testdata/control_images/expected_atlas_sorting2/expected_atlas_sorting2_mask.png
  43. BIN tests/testdata/control_images/expected_atlas_sorting2/quantal/expected_atlas_sorting2.png
  44. BIN tests/testdata/control_images/expected_atlas_two_maps1/default/expected_atlas_two_maps1.png
  45. BIN tests/testdata/control_images/expected_atlas_two_maps1/{precise → }/expected_atlas_two_maps1.png
  46. BIN tests/testdata/control_images/expected_atlas_two_maps1/expected_atlas_two_maps1_mask.png
  47. BIN tests/testdata/control_images/expected_atlas_two_maps2/default/expected_atlas_two_maps2.png
  48. BIN tests/testdata/control_images/expected_atlas_two_maps2/{precise → }/expected_atlas_two_maps2.png
  49. BIN tests/testdata/control_images/expected_atlas_two_maps2/expected_atlas_two_maps2_mask.png
  50. BIN .../expected_composerattributetable_columnwidth/expected_composerattributetable_columnwidth_mask.png
  51. BIN .../expected_composerattributetable_headersonly/expected_composerattributetable_headersonly_mask.png
  52. BIN ...rol_images/expected_composerattributetable_render/expected_composerattributetable_render_mask.png
  53. BIN tests/testdata/control_images/expected_composereffects_blend/expected_composereffects_blend_mask.png
  54. BIN tests/testdata/control_images/expected_composermap_grid/expected_composermap_grid_mask.png
  55. BIN tests/testdata/control_images/expected_composermap_render/expected_composermap_render_mask.png
  56. BIN ...l_images/expected_composermap_rotatedannotations/expected_composermap_rotatedannotations_mask.png
  57. BIN ...xpected_composerpicture_resize_frametoimage/expected_composerpicture_resize_frametoimage_mask.png
  58. BIN .../testdata/control_images/expected_composerrotation_label/expected_composerrotation_label_mask.png
  59. BIN ...a/control_images/expected_composerscalebar_singlebox/expected_composerscalebar_singlebox_mask.png
  60. BIN ...ages/expected_composerscalebar_singlebox_alpha/expected_composerscalebar_singlebox_alpha_mask.png
  61. BIN tests/testdata/control_images/expected_composerutils_drawtext_pos/anomaly_win7.png
  62. BIN ...a/control_images/expected_composerutils_drawtext_pos/expected_composerutils_drawtext_pos_mask.png
  63. BIN tests/testdata/control_images/expected_composerutils_drawtext_rect/anomaly_win7.png
  64. BIN ...control_images/expected_composerutils_drawtext_rect/expected_composerutils_drawtext_rect_mask.png
  65. BIN ...ta/control_images/expected_inverted_polys_graduated/default/expected_inverted_polys_graduated.png
  66. BIN ...ntrol_images/expected_inverted_polys_graduated/{precise → }/expected_inverted_polys_graduated.png
  67. BIN ...tdata/control_images/expected_inverted_polys_graduated/expected_inverted_polys_graduated_mask.png
@@ -0,0 +1,86 @@
#!/usr/bin/env python

# Generates (or updates) a unit test image mask, which is used to specify whether
# a pixel in the control image should be checked (black pixel in mask) or not (white
# pixel in mask). For non black or white pixels, the pixels lightness is used to
# specify a maximum delta for each color component

import os
import sys
import argparse
from PyQt4.QtCore import *
from PyQt4.QtGui import *
import struct

def error ( msg ):
print msg
sys.exit( 1 )

def colorDiff( c1, c2 ):
redDiff = abs( qRed( c1 ) - qRed( c2 ) )
greenDiff = abs( qGreen( c1 ) - qGreen( c2 ) )
blueDiff = abs( qBlue( c1 ) - qBlue( c2 ) )
alphaDiff = abs( qAlpha( c1 ) - qAlpha( c2 ) )
return max( redDiff, greenDiff, blueDiff, alphaDiff )

def updateMask(control_image_path, rendered_image_path, mask_image_path):
control_image = QImage( control_image_path )
if not control_image:
error('Could not read control image {}'.format(control_image_path))

rendered_image = QImage( rendered_image_path )
if not rendered_image:
error('Could not read rendered image {}'.format(rendered_image_path))
if not rendered_image.width() == control_image.width() or not rendered_image.height() == control_image.height():
error('Size mismatch - control image is {}x{}, rendered image is {}x{}'.format(control_image.width(),
control_image.height(),
rendered_image.width(),
rendered_image.height()))

#read current mask, if it exist
mask_image = QImage( mask_image_path )
if mask_image.isNull():
print 'Mask image does not exist, creating'
mask_image = QImage( control_image.width(), control_image.height(), QImage.Format_ARGB32 )
mask_image.fill( QColor(0,0,0) )

#loop through pixels in rendered image and compare
mismatch_count = 0
width = control_image.width()
height = control_image.height()
linebytes = width * 4
for y in xrange( height ):
control_scanline = control_image.constScanLine( y ).asstring(linebytes)
rendered_scanline = rendered_image.constScanLine( y ).asstring(linebytes)
mask_scanline = mask_image.scanLine( y ).asstring(linebytes)

for x in xrange( width ):
currentTolerance = qRed( struct.unpack('I', mask_scanline[ x*4:x*4+4 ] )[0] )

if currentTolerance == 255:
#ignore pixel
continue

expected_rgb = struct.unpack('I', control_scanline[ x*4:x*4+4 ] )[0]
rendered_rgb = struct.unpack('I', rendered_scanline[ x*4:x*4+4 ] )[0]
difference = colorDiff( expected_rgb, rendered_rgb )

if difference > currentTolerance:
#update mask image
mask_image.setPixel( x, y, qRgb( difference, difference, difference ) )
mismatch_count += 1

if mismatch_count:
#update mask
mask_image.save( mask_image_path, "png" );
print 'Updated {} pixels'.format( mismatch_count )

parser = argparse.ArgumentParser() #OptionParser("usage: %prog control_image rendered_image mask_image")
parser.add_argument('control_image')
parser.add_argument('rendered_image')
parser.add_argument('mask_image')
args = parser.parse_args()


updateMask(args.control_image, args.rendered_image, args.mask_image)

@@ -265,6 +265,17 @@ bool QgsRenderChecker::compareImages( QString theTestName,
theTestName + "_result_diff.png";
myDifferenceImage.fill( qRgb( 152, 219, 249 ) );

//check for mask
QString maskImagePath = mExpectedImageFile;
maskImagePath.chop( 4 ); //remove .png extension
maskImagePath += "_mask.png";
QImage* maskImage = new QImage( maskImagePath );
bool hasMask = !maskImage->isNull();
if ( hasMask )
{
qDebug( "QgsRenderChecker using mask image" );
}

//
// Set pixel count score and target
//
@@ -338,6 +349,7 @@ bool QgsRenderChecker::compareImages( QString theTestName,
mReport += "<font color=red>Expected image and result image for " + theTestName + " are different dimensions - FAILING!</font>";
mReport += "</td></tr>";
mReport += myImagesString;
delete maskImage;
return false;
}

@@ -348,29 +360,42 @@ bool QgsRenderChecker::compareImages( QString theTestName,

mMismatchCount = 0;
int colorTolerance = ( int ) mColorTolerance;
for ( int x = 0; x < myExpectedImage.width(); ++x )
for ( int y = 0; y < myExpectedImage.height(); ++y )
{
for ( int y = 0; y < myExpectedImage.height(); ++y )
const QRgb* expectedScanline = ( const QRgb* )myExpectedImage.constScanLine( y );
const QRgb* resultScanline = ( const QRgb* )myResultImage.constScanLine( y );
const QRgb* maskScanline = hasMask ? ( const QRgb* )maskImage->constScanLine( y ) : 0;
QRgb* diffScanline = ( QRgb* )myDifferenceImage.scanLine( y );

for ( int x = 0; x < myExpectedImage.width(); ++x )
{
QRgb myExpectedPixel = myExpectedImage.pixel( x, y );
QRgb myActualPixel = myResultImage.pixel( x, y );
if ( mColorTolerance == 0 )
int maskTolerance = hasMask ? qRed( maskScanline[ x ] ) : 0;
int pixelTolerance = qMax( colorTolerance, maskTolerance );
if ( pixelTolerance == 255 )
{
//skip pixel
continue;
}

QRgb myExpectedPixel = expectedScanline[x];
QRgb myActualPixel = resultScanline[x];
if ( pixelTolerance == 0 )
{
if ( myExpectedPixel != myActualPixel )
{
++mMismatchCount;
myDifferenceImage.setPixel( x, y, qRgb( 255, 0, 0 ) );
diffScanline[ x ] = qRgb( 255, 0, 0 );
}
}
else
{
if ( qAbs( qRed( myExpectedPixel ) - qRed( myActualPixel ) ) > colorTolerance ||
qAbs( qGreen( myExpectedPixel ) - qGreen( myActualPixel ) ) > colorTolerance ||
qAbs( qBlue( myExpectedPixel ) - qBlue( myActualPixel ) ) > colorTolerance ||
qAbs( qAlpha( myExpectedPixel ) - qAlpha( myActualPixel ) ) > colorTolerance )
if ( qAbs( qRed( myExpectedPixel ) - qRed( myActualPixel ) ) > pixelTolerance ||
qAbs( qGreen( myExpectedPixel ) - qGreen( myActualPixel ) ) > pixelTolerance ||
qAbs( qBlue( myExpectedPixel ) - qBlue( myActualPixel ) ) > pixelTolerance ||
qAbs( qAlpha( myExpectedPixel ) - qAlpha( myActualPixel ) ) > pixelTolerance )
{
++mMismatchCount;
myDifferenceImage.setPixel( x, y, qRgb( 255, 0, 0 ) );
diffScanline[ x ] = qRgb( 255, 0, 0 );
}
}
}
@@ -379,6 +404,7 @@ bool QgsRenderChecker::compareImages( QString theTestName,
//save the diff image to disk
//
myDifferenceImage.save( myDiffImageFile );
delete maskImage;

//
// Send match result to debug
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.
Diff not rendered.

0 comments on commit 854c0b8

Please sign in to comment.
You can’t perform that action at this time.