Bugs in Folding Augmentation

[proofconstruction]

@kwcckw I found some bugs in folding.py, can you take a look?

Bugs

1. If image is less than 10x10, fold_y_shift_min and fold_y_shift_max in folding.py will become 0 after multiplication by a number in (0.1,0.2) due to the int cast, which makes fold_y_shift on the following line fail trying to generate a random from an empty range. This can be resolved by restricting images to be a minimum of 10x10.
2. It seems like fold_y_shift is sometimes 0 for similar reasons, which makes the call to img_warped[:-fold_y_shift ,:,:] on line 158 fail. It tries to take an empty slice of the array using :-fold_y_shift, which makes the img_fuse[cys:cye,cxs:cxe,:] call on that same line fail with a ValueError: could not broadcast input array from shape (0,14,3) into shape (95,14,3) (for example - the real numbers are different every time, but the first value of the triple is always 0 because no elements were selected by the array slice.)
3. An OpenCV error occurs in four_point_transform(), but I haven't been able to track down why yet. Here's the stack trace:

============================== FAILURES ===============================
________________________ test_folding_pipeline ________________________

random_image = array([[[168, 217, 242],
        [144, 158,  82],
        [194,  42, 206],
        [ 89, 208,  93],
        [ 23, 197,...57],
        [255, 235,  18],
        [161,  34,  28],
        [231, 253, 229],
        [ 78, 119, 244]]], dtype=uint8)
folding_pipeline = ink_phase = AugmentationSequence([
])

paper_phase = AugmentationSequence([
])

post_phase = AugmentationSequence([
])...Pipeline(ink_phase, paper_phase, post_phase, ink_color_range=(0, 0), paper_color_range=(255, 255), rotate_range=(0, 0))

    def test_folding_pipeline(random_image, folding_pipeline):
>       crappified = folding_pipeline.augment(random_image)

test_folding_pipeline.py:22: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
augraphy/base/augmentationpipeline.py:144: in augment
    self.post_phase(data)
augraphy/base/augmentationsequence.py:28: in __call__
    augmentation(data)
augraphy/augmentations/folding.py:224: in __call__
    image_fold = self.apply_folding(image_fold, ysize, xsize, self.gradient_width, self.gradient_height, self.fold_noise)
augraphy/augmentations/folding.py:205: in apply_folding
    img_fold_l = self.warp_fold_left_side(img, ysize, fold_noise, fold_x, fold_width_one_side, fold_y_shift)
augraphy/augmentations/folding.py:99: in warp_fold_left_side
    img_warped = self.four_point_transform(img_crop,source_pts,destination_pts, cxsize, cysize+fold_y_shift)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Folding(fold_count=2, fold_noise=0.1, gradient_width=(0.1, 0.2), gradient_height=(0.01, 0.02),p=0.5)
image = array([], shape=(12, 0, 3), dtype=uint8)
pts = array([[ 0.,  0.],
       [ 0., 12.],
       [ 0., 12.],
       [ 0.,  0.]], dtype=float32)
dst = array([[ 0.,  0.],
       [ 0., 12.],
       [ 0., 12.],
       [ 0.,  0.]], dtype=float32)
xs = 0, ys = 12

    def four_point_transform(self, image,pts,dst, xs,ys):
        M = cv2.getPerspectiveTransform(pts, dst)
>       img_warped = cv2.warpPerspective(image, M, (xs, ys))
E       cv2.error: OpenCV(4.5.3) /tmp/pip-req-build-3umofm98/opencv/modules/imgproc/src/imgwarp.cpp:3144: error: (-215:Assertion failed) _src.total() > 0 in function 'warpPerspective'

augraphy/augmentations/folding.py:49: error
======================= short test summary info =======================
FAILED test_folding_pipeline.py::test_folding_pipeline - cv2.error: ...
========================== 1 failed in 0.32s ==========================

Testing

You can test this yourself by making test_folding_augmentation.py in the project directory with the following contents:

import random
import numpy as np
import pytest

from augraphy import *

@pytest.fixture
def folding_pipeline():
    return AugraphyPipeline([], [], [Folding()])

@pytest.fixture
def random_image():
    xdim = random.randint(1,500)
    ydim = random.randint(1,500)
    return np.random.randint(
        low=0,
        high=255,
        size=(xdim,ydim,3),
        dtype=np.uint8)

def test_folding_augmentation(random_image, folding_pipeline):
    crappified = folding_pipeline.augment(random_image)

and running pytest test_folding_augmentation.py a bunch of times (it needs to be run several times because this bug doesn't always appear). You can make the first two bugs appear very reliably by setting xdim and ydim to numbers less than 10.

For easy testing you can run for i in {1..20}; do pytest test_folding_augmentation.py; done;

[kwcckw]

Thanks, i will test it again the patch it in the next update.

[proofconstruction]

I merged PR #57 which seems to work fine and not throw bugs 1 and 2 above, but also we now only test with images larger than 30x30.

However , we are still getting an error in some runs of Folding.

[kwcckw]

I merged PR #57 which seems to work fine and not throw bugs 1 and 2 above, but also we now only test with images larger than 30x30.

However , we are still getting an error in some runs of Folding.

Thanks for testing it out . I couldn't get a similar error but I'm getting a different error instead. Could you try again to replace the apply_folding function with this and check whether you facing error with image size < 30 and >1?

# Apply perspective transform 2 times and get single folding effect
def apply_folding(self, img, ysize, xsize, gradient_width, gradient_height, fold_noise):
     

    min_fold_x = min(np.ceil(gradient_width[0] * xsize),xsize).astype('int')
    max_fold_x = min(np.ceil(gradient_width[1] * xsize),xsize).astype('int')
    fold_width_one_side = int(random.randint(min_fold_x, max_fold_x)/2)      # folding width from left to center of folding, or from right to center of folding
    
    # test for valid folding center line
    if (xsize-fold_width_one_side-1) < (fold_width_one_side+1):
        print("Folding augmentation is not applied, please increase image size")
        return img
    fold_x = random.randint(fold_width_one_side+1, xsize-fold_width_one_side-1) # center of folding
    
    fold_y_shift_min = min(np.ceil(gradient_height[0] * ysize),ysize).astype('int')
    fold_y_shift_max = min(np.ceil(gradient_height[1] * ysize),ysize).astype('int')
    fold_y_shift = random.randint(fold_y_shift_min, fold_y_shift_max) # y distortion in folding (support positive y value for now)
    
    # test for valid folding width and folding distortion
    if (fold_width_one_side != 0) and (fold_y_shift !=0) :
        img_fold_l = self.warp_fold_left_side(img, ysize, fold_noise, fold_x, fold_width_one_side, fold_y_shift)
        img_fold_r = self.warp_fold_right_side(img_fold_l, ysize, fold_noise, fold_x, fold_width_one_side, fold_y_shift)
        return img_fold_r
    else:
        if (fold_width_one_side != 0):
            print("Folding augmentation is not applied, please increase gradient width or image size")
        else:
            print("Folding augmentation is not applied, please increase gradient height or image size")    
        return img

[proofconstruction]

Fixed with PR #59