Patches Visualization

[nontas]

This PR adds functionality for visualizing a set of patches that are extracted from an image.

I. Patches representation

All the changes are made based on the assumption that patches are represented in one of the two formats that are returned by the extract_patches() method of Menpo's Image class, i.e.

• A numpy.array with shape (n_centers, n_offsets, n_channels, patch_height, patch_width)
• A list of n_centers * n_offsets Image objects, each of which has shape (n_channels, patch_height, patch_width)

where n_centers is the number of patches (points) and n_offsets is the number of offsets per patch. Generally it is more efficient to use the first representation, that's why a convert_patches_list_to_single_array(patches_list, n_centers) method is defined in menpo.image.base. I also changed the default value of as_single_array to True in extract_patches() and extract_patches_around_landmarks() methods.

II. Set patches

In order to facilitate the patches visualization, I defined methods that are able to set patches to the correct locations of an image. Specifically,

1. I defined a Cython set_patches() method, similar to the existing extract_patches() one.
2. The method is added to the old menpo.image.extract_patches.pyx file, which is now renamed to menpo.image.patches.pyx.
3. The set_patches(patches, patch_centers, offset, offset_index) and set_patches_around_landmarks(patches, group, label, offset, offset_index) methods now exist in all image classes, i.e. Image, MaskedImage and BooleanImage. offset is a single offset value ((1, 2 numpy.array or list or tuple) that gets added to patch_centers. offset_index is the offset index value of patches.
4. The way that the patches were constructed through build_mask_around_landmarks() was different than extract_patches() and extract_patches_around_landmarks(). Thanks to @patricksnape this is now made consistent.

III. Visualize patches

A patches numpy.array or list can be visualized in two ways:

III.A. view_patches() from menpo.image

The basic arguments of this method are patches and patch_centers. The user can also pass in a list with the patches indices to get visualized as well as select the patches offset. The method can optionally visualize the patches centers (PointCloud) as well as draw the bounding boxes around the patches. Finally, the background colour can optionally be either black or white.

III.B. visualize_patches() widget from menpo.visualize

The widget has a similar behaviour to the rest of the widgets. It includes options related to the channels and the renderer. However, it has an extra tab of options related to the patches. In there, the user can define the following:

• The patches indices to be visualized using a valid Python slicing command, e.g. ::3, :3, 1:10:2, 1, 2, 3, [1, 2, 3], range(10), range(2, 10) etc.
• Offset index
• Background colour, bounding boxes options, render patches and centers checkboxes

IV. Example code

%matplotlib inline
import menpo.io as mio
from menpo.image import view_patches
from menpo.visualize import visualize_patches
import matplotlib.pyplot as plt

im = mio.import_builtin_asset.lenna_png()
patches = im.extract_patches_around_landmarks(patch_shape=(21, 12))

# Visualize patches with various options
plt.subplot(131)
view_patches(patches, im.landmarks['LJSON'].lms, render_patches=True,
             background='white', render_centers=False, 
             render_patches_bboxes=False, figure_size=(16, 14))

plt.subplot(132)
view_patches(patches, im.landmarks['LJSON'].lms, render_patches=True,
             background='black', render_centers=True, 
             render_patches_bboxes=True, figure_size=(16, 14))

plt.subplot(133)
view_patches(patches, im.landmarks['LJSON'].lms, render_patches=False,
             background='white', render_centers=True, 
             render_patches_bboxes=True, figure_size=(16, 14))

# Widget
visualize_patches(patches, im.landmarks['LJSON'].lms, style='coloured')

V. Open questions

There are still some open questions regarding the logic that I followed, for which I need your feedback:

1. Does the patches representation make sense? Should we define a PathBasedImage class? (I have already discussed these with @patricksnape )
2. Does it make sense to have the view_patches() method in menpo.image or should it be moved to menpo.visualize?
3. Does this solution make it easy to visualize things from menpofit, for example the response maps of CLMs?

[patricksnape]

Nooooo, why is this patch_shape now? If we change patch_size to patch_shape then we need to do the same in menpofit!

[nontas]

True... However, I think we should be consistent on the naming of this argument. So we need to choose either patch_size or patch_shape (I prefer it) and use it everywhere. I will make a PR in menpofit that will add a widget for patch-based appearance models visualization. So the PR can also include renaming all patch_size occurrences to patch_shape.

[nontas]

Following our discussion with @patricksnape and @jabooth , it is agreed that we always use patch_shape and we make it consistent throughout menpo and menpofit.

[nontas]

Just an update: menpo/menpofit#72 now defines the respective patch-based widgets in menpofit, so if you guys are happy, the two should be merged simultaneously.

[patricksnape]

I guess we get this in and then split out the widgets?

[nontas]

After discussing with @jabooth

• create_patches_image() is now made invisible
• view_patches is renamed to view_patches_nowidget and moved to menpo.visualize

Tell me you're happy. After this we'll split out the widgets (and visualize as well??)

[patricksnape]

Just the widgets - visualize will stay because you can still view without a notebook. I'm happy with this and really need the patch_shape change! +1

[jabooth]

underscore this name too, no point making it public API too early.

[nontas]

done

[jabooth]

Just that one function to hide then we can get this in - +1