New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

warp_to_mask, warp_to_shape, fast resizing of images #369

Merged
merged 46 commits into from Sep 13, 2014

Conversation

Projects
None yet
4 participants
@jabooth
Member

jabooth commented May 21, 2014

This PR changes the warping interface in Menpo.

We've always had warp_to, a method that took a boolean mask and used it to warp Image s and MaskedImage s.

Unfortunately, this wasn't well tested, and suffered from a number of shortcomings:

  1. warping an Image with anything except an all-true mask actually led to an Exception. This meant that it wasn't possible to fit an Image to an AAM - something that doesn't make any sense as a new image to fit has no need for a mask.
  2. Even though the above was true we were using this method when rescaling Images. It just so happened that because the mask was all true, it worked.
  3. The behavior was complex. A kwarg (warp_mask) controlled if the mask on a masked image should be warped or not. By default it was false, so the mask on a masked image actually had no effect on the warping.
  4. skimage.transform.resize was over twice as fast as our resize on a plain Image, because they have a custom Cython interpolator for affine warps that is simply faster than map_coordinates. resize is used a lot - every fitting method starts by getting the image to be fitted to be of the same units as the model, which requires a resize. Before this PR, resizing at the start of the SDM fitting loop was dominating, taking about 40% of the time for a fit.

Proposal

This PR fixes this by splitting warp_to into two methods - warp_to_mask and warp_to_shape. warp_to_mask behaves just like warp_to did, with a few changes:

  1. The type of return for both Image.warp_to_mask and MaskedImage.warp_to_mask is always
    MaskedImage. This means AAM can call warp_to_mask on an input image and know for sure the result is masked, fixing a longstanding bug.
  2. warp_to_mask on a MaskedImage now always warps the mask. This is equivalent to forcing the warp_mask=True kwarg on warp_to. This is much more natural. If you don't care about the mask, you should efficiently convert the MaskedImage to an Image and call warp_to_mask on that (more about that in a sec). Note that this is the biggest breaking change in the PR. We need to analyse where we are currently calling warp_to and decide how it should be handled now.

warp_to_shape is a new warping method. It always returns the same type as what it is called on. This makes it much more friendly to be called from methods like resize on the image package.

Instead of taking a template mask this variant takes a template shape. This matches exactly the interface to skimage.transform.warp and in fact actually calls that method (basically, it's a thin wrapper around their method which handles converting our transform types into callables they expect). This can be much more efficient than the masking case (and indeed it is, it gives us an over 2x performance increase in resize).

To do

The work isn't quite done, but I wanted to put this up to get feedback. Still outstanding we have

  • Fix PWA issues that we are having in LK
  • Allow boolean ndarray as well as BooleanImage for mask in warp_to_mask
    I would argue it's cleaner to just use BooleanImage for clarity, unless there are strong objections?
  • Adjust kwargs on warp_to_image to match warp_to_mask
  • Decide on how interpolator kwarg should be handled
interpolator=self.interpolator)
# TODO should the template be a mask or a shape? warp_to_shape here
warped_image = image.warp_to_mask(self.template.mask, self.transform,
interpolator=self.interpolator)

This comment has been minimized.

@jabooth

jabooth May 21, 2014

Member

@jalabort what exactly is this warp for? Is the template really a mask, or are you just using it for shape? Could replace with warp_to_shape if so...

This comment has been minimized.

@jalabort

jalabort May 26, 2014

Member

This is a warp onto the mask of reference frame of the AAM which atm is always a boolean image defining the facial pixels on the reference frame. This is equivalent to the:

IWxp = image.warp_to ...

Of the LK methods...

jabooth added some commits May 22, 2014

Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fitmultilevel/aam/builder.py
Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fitmultilevel/aam/builder.py
	menpo/image/base.py
	menpo/image/boolean.py
Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/image/base.py
	menpo/io/input/image.py
	menpo/io/test/io_import_test.py
@jabooth

This comment has been minimized.

Member

jabooth commented Aug 26, 2014

Just merged in master to bring this roughly up to date. Not a huge amount of work left in getting this ready, just:

  • Figure out why some tests are failing
  • Update notebooks and test extensively

@jabooth jabooth added the in progress label Aug 27, 2014

@jabooth jabooth self-assigned this Aug 27, 2014

jabooth added some commits Aug 28, 2014

Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fit/lucaskanade/test/residual_test.py
	menpo/io/test/io_import_test.py
Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fit/fittingresult.py
	menpo/fit/lucaskanade/test/residual_test.py
	menpo/fitmultilevel/base.py
	menpo/fitmultilevel/fittingresult.py
	menpo/io/test/io_import_test.py
Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fitmultilevel/aam/builder.py
	menpo/fitmultilevel/builder.py
	menpo/fitmultilevel/clm/builder.py
	menpo/fitmultilevel/sdm/base.py
	menpo/fitmultilevel/test/aam_builder_test.py
	menpo/fitmultilevel/test/aam_fitter_test.py
	menpo/fitmultilevel/test/clm_builder_test.py
	menpo/fitmultilevel/test/clm_fitter_test.py

jabooth added some commits Sep 6, 2014

Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fitmultilevel/clm/builder.py
Merge remote-tracking branch 'upstream/master' into warp_to_mask
Conflicts:
	menpo/fitmultilevel/sdm/trainer.py
@jalabort

This comment has been minimized.

Member

jalabort commented Sep 7, 2014

I had a look at the test failing here...

In my mac:

test_alternating_fa changes from 9.90404... to 9.90421361884
test_projectout_fa changes from 240.78115... to 240.781168036

I don't know what's causing this and I'm not sure is relevant... AAMs on my notebooks seemed to function well ;-)

On another note, Travis is telling us we've got a possible division by 0 on the ES features that we are not taking care of... Maybe we should just add a check for this...

@jabooth

This comment has been minimized.

Member

jabooth commented Sep 7, 2014

@jalabort it's not surprising that we might get slight numerical differences - if you are happy with the performance I say we get it in! It's only been 109 days since I put this PR up ;)

Are we happy with the change to importing Image instead of MaskedImage by default?

As for the errors they have been around for a little while, I'll make an issue to track it, thanks!

@jabooth jabooth referenced this pull request Sep 7, 2014

Closed

Fix nosetest runtime errors #436

@jabooth

This comment has been minimized.

Member

jabooth commented Sep 9, 2014

@patricksnape has discovered that skimage's warp method (which this PR relies on) is very opinionated about the range of allowable outputs - namely that it clips them to [0 - 1]. This may explain the few numerical differences we are seeing.

This PR will need to be adapted to use a modified version of the scikit image warp function before we bring it in.

@patricksnape

This comment has been minimized.

Contributor

patricksnape commented Sep 12, 2014

I ran a few of the benchmarks and everything looks OK. We do slightly worse on LFPW with IGO and 1 level, but that could be just numerical differences. We still get over 90% for HOGs on LFPW with 3 levels.

+1

patricksnape and others added some commits Sep 12, 2014

@nontas

This comment has been minimized.

Member

nontas commented Sep 13, 2014

If travis gives permission, then this is ready
+1

@jabooth

This comment has been minimized.

Member

jabooth commented Sep 13, 2014

Travis loves it! Thanks @nontas

jabooth added a commit that referenced this pull request Sep 13, 2014

Merge pull request #369 from jabooth/warp_to_mask
warp_to_mask, warp_to_shape, fast resizing of images

@jabooth jabooth merged commit 4495102 into menpo:master Sep 13, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details

@jabooth jabooth removed the in progress label Sep 13, 2014

@jabooth jabooth deleted the jabooth:warp_to_mask branch Sep 13, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment