New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batched Transforms And OutOfMask Sampling #563

Merged
merged 20 commits into from Mar 26, 2015

Conversation

Projects
None yet
3 participants
@patricksnape
Contributor

patricksnape commented Mar 10, 2015

Original

Allow for large warping and account for an instability that can occur in TPS warps. See commit messages for more details.

Edit

This PR has become a bit more heavy handed, basically adding a bunch of stuff that I need.

  1. Add a sample method to images for sampling from arbitrary floating point locations.
  2. Change how warping masked images works. Now, warping a masked image must mean that all sampled values land inside the mask. If you want the previous behaviour of the 'union' of masks, just warp the mask, warp the image as unmasked, then convert the warped image back to a masked image and set the warped mask.
  3. Add batching to transforms. Now all transforms can be batch applied. This is useful for keeping memory usage low. It extends the original goal of batching for warping to batching for ALL transforms.
  4. Optimise constrain_mask_to_landmarks to only consider the bounding box of the landmarks, which thus ends up transforming many less points. Constraining the BreakingBad image goes from 7s to 700ms on my machine with this change.
  5. Allow batching for warp_to_mask, warp_to_shape and constrain_mask_to_landmarks
  6. Add new kwarg to as_unmasked which allows setting the value of the unmasked pixels. This is useful for 'baking' the mask for visualization, without needing to use Matplotlib.
  7. Allow for inexact TPS solution by adding a small regularizer to the inverse.

patricksnape added some commits Mar 10, 2015

Allow warping of large images using batches
The TPS cdist function creates enormous symmetric matrices
for large images (since there are a lot of indices to sample
inside large images). Therefore, this commit allows an optional
kwarg batch_size for warping, whereby the points to warp get
passed in smaller batches. By default, this is not used.
Account for instability in TPS inverse
If the target has points that are nearly coincident, the coefficients
matrix is rank deficient, and therefore not invertible. Therefore, we
only take the inverse on the full-rank matrix and drop any singular
values that are less than a very small value (close to zero).
@jalabort

This comment has been minimized.

Member

jalabort commented Mar 10, 2015

Did it solve the weird warpings you were getting?

@patricksnape

This comment has been minimized.

Contributor

patricksnape commented Mar 10, 2015

No :( But I just copied it from Yuxiang, so I guess it fixed it for
him. It did make them better, but not perfect. But it's fine because I
can use the other way.

On 3/10/15, Joan Alabort-i-Medina notifications@github.com wrote:

Did it solve the weird warpings you were getting?


Reply to this email directly or view it on GitHub:
#563 (comment)

@jalabort

This comment has been minimized.

Member

jalabort commented Mar 10, 2015

OK, good. I think in general this means that the TPS warp is not exact any more, i.e. landmark correspondences are not strictly kept... You could also try using a slightly bigger min_singular_val for your use case.

patricksnape added some commits Mar 25, 2015

Add batch_size to transform.apply
This allows you to batch the input to a transform, which is more
general than the batched input for image warping that I had
before. This is particularly useful for transforms that have large
intermediate data structures such as PWA and TPS.
Change how sampling is done for warping
Now, sampling is still batched, but there is an explicit sampling
method. Also, batching is now removed as it is done by the
transforms themselves.
Fix small bug whereby extra arguments are passed to BooleanImage
Just abuse kwargs. order shouldn't be specified to BooleanImage,
but since we reuse the Image superclass implementation it is,
therefore, just ignore the order parameter by abusing kwargs.
Change default 0.0 to False for BooleanImage
Default cval was 0.0, might as well be False, as this is more
accurate for the boolean image data type.
Breaking change for mask warping
Given the new sampling behaviour, there is now a new definition
for the behaviour of warping masked images. It will **only**
warp if the target mask is True everywhere that the reference
mask wants to sample. Otherwise, it will throw an
OutOfMaskSampleError, which will fix problems for things like
building AAMs with masked Images.

If you want to sample the whole image, you as_unmasked.
If you want to warp the mask, use warp_to_shape on the mask, warp
the unmasked image, then create a masked image and set the
warped mask on the warped image.
Missing setting path on MaskedImage warp_to_shape
Also, only set landmarks if it had landmarks
Update constrain_mask_to_landmarks
Only consider those points that are inside the bounding
box of the landmarks. This is a massive performance improvement
for large images like breaking bad with a small masked region.
Add batch_size to constrain_mask_to_landmarks
Also, fix but about catching and throwing the
TriangleContainmentError in PWA. Simply needed to override the
_apply_batched method for PWA.
Fix TriangleContainmentError
Missed the case whereby all the points are INSIDE the bounding
box, aka the landmarks ARE a box. This should set all points to
true.

Also, the bounds are inclusive on the top end, so we add +1 to
the max bounds to be correct. Added tests for these cases
and checked on master.
Add fill to as_unmasked
This allows 'baking' of the mask. Useful for visualization.
@patricksnape

This comment has been minimized.

Contributor

patricksnape commented Mar 25, 2015

See description, I have edited it to include all the new changes I've made. I've tried to add tests for any of the new functionality I added.

patricksnape added some commits Mar 25, 2015

Alignment.__init__(self, source, target)
if self.n_dims != 2:
raise ValueError('TPS can only be used on 2D data.')
if kernel is None:
kernel = R2LogR2RBF(source.points)
self._min_singular_val = min_singular_val

This comment has been minimized.

@jabooth

jabooth Mar 26, 2015

Member

@patricksnape I think this would be better non '_' - imagine users would want to be able to check this parameter after constructing warps in the future

Change min_singular_value to non-private
Loses underscore so that is a tunable parameter after
the transform has been built.
sampled_pixels : (`n_points`, `n_channels`) `ndarray`
The interpolated values taken across every channel of the image.
"""
from menpo.shape import PointCloud

This comment has been minimized.

@jabooth

jabooth Mar 26, 2015

Member

@patricksnape do we need this import inline? maybe if we do add a comment to clarify there is a circular import issue

@jabooth

This comment has been minimized.

Member

jabooth commented Mar 26, 2015

+1

jabooth added a commit that referenced this pull request Mar 26, 2015

@jabooth jabooth merged commit 74571e8 into menpo:master Mar 26, 2015

2 of 3 checks passed

continuous-integration/appveyor AppVeyor build failed
Details
clahub All contributors have signed the Contributor License Agreement.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@jabooth jabooth removed the in progress label Mar 26, 2015

@jabooth jabooth deleted the patricksnape:large_warps branch Mar 26, 2015

@patricksnape patricksnape changed the title from TPS Updates to TPS Updates And OutOfMask Sampling May 13, 2015

@patricksnape patricksnape changed the title from TPS Updates And OutOfMask Sampling to Batched Transforms And OutOfMask Sampling May 13, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment