Add a tile size parameter for TileSource.tileIterator

[cdeepakroy]

This is to provide the ability to use a custom tile size when iterating through a whole slide image at a desired magnification.

Example usage:

ts = large_image.getTileSource(wsi_path)
for tile in ts.tileIterator(scale={'magnification': 20}, 
                            tile_size=(tile_height, tile_width)):
    pass

[manthey]

Further, this could support overlap:

tile_size={
  'width': width.
  'height': height,
  'overlap_x': overlap_x,
  'overlap_y': overlap_y
}

where overlap_x and overlap_y would be non-negative integers. The overlap would be applied equally on the opposite sides. Or, would overlap be better to be customizable in all four directions?

[cdeepakroy]

I think equally on both sides would be good enough. I would suggest using a separate argument for overlap

ts = large_image.getTileSource(wsi_path)
for tile in ts.tileIterator(scale={'magnification': 20}, 
                            tile_size={'width': width, 'height': height},
                            tile_overlap={'x': None, 'y': None}):
    pass

[cdeepakroy]

cc: @cooperlab

[cooperlab]

Equal on both sides is good. I also like splitting out the arguments.

[manthey]

Great. Will do.

[manthey]

So if we specify a tile size and overlap, is the size excluding the overlap or including the overlap?

[cdeepakroy]

The size would exclude the overlap. Visualize a sliding window moved across the image from left-to-right and top-to-bottom --

• tile_size specifies the size of the window and
• tile_overlap specifies the amount of overlap of adjacent windows (along x and y dirs).

It would be nice if you could also make getSingleTile receive the same arguments -- tile_size and tile_overlap.

[manthey]

One virtue of everything going through _tileIteratorInfo internally is that getSingleTile will get these options without additional work.

[manthey]

@cdeepakroy I've got this working on my machine and am adding tests.

When you ask for overlap, what would you like to occur at the edges? Currently, my code adds overlap but will always stop at the edge of the image. So, for instance if you ask for tiles that are 256x256 with overlap in each direction of 16. in the center of the image, you'll get a tile that is 288x288, and in the upper left corner you get a tile that is 272x272. Should this be reported in the tile information (e.g., 'overlap': {'left': <>, 'top': <>, 'right': <>, 'bottom': <>})? Or should we add empty pixels so the overlap is always the same?

[manthey]

And, if you ask for a region that isn't the whole image, should the overlap be limited to that region, or should it extend into the rest of the image? For instance, if you ask for the region that is from (100, 150) to (400, 700) -- a region that is 300 x 550 pixels in size, is there no overlap at 100, 150, or does the overlap extend into the image beyond the originally specified region?

[cooperlab]

@manthey @cdeepakroy I'm thinking that if you ask for a tile that is 256 x 256 then you should get a tile that size instead of something larger (even with non-zero overlap). You will have a shorter stride between tile centers but that's OK.

Re edges - I may want to know what parts of the tile are empty. I would just fill to the prescribed size if you run off the edge, just like we do with ordinary tiling.

[manthey]

Currently, if we run off the edge of a requested region (which defaults to the whole image), the result is cropped. We could make this an option. If so, the option could be something like edge which would default to crop or take a color that would be used to fill the region.

If we aren't cropping and the requested region is not the whole image, I would think that the fill area should be the image outside of the region rather than the fill color.

[cooperlab]

OK - I didn't know that. I think we can live with cropping now. I would just look at overlap as a special case where the stride is reduced then. Shouldn't be any special considerations in that case.

[manthey]

I've switched the PR so that tile size includes the overlap. We can always add a no-crop option in the future.

[cdeepakroy]

@cooperlab After reading through our discussion here, what do you think is more preferrable -- specifying overlap or stride (CNN style)?

@manthey This confuses me "So, for instance if you ask for tiles that are 256x256 with overlap in each direction of 16. in the center of the image, you'll get a tile that is 288x288, and in the upper left corner you get a tile that is 272x272."

Why would the tile size be different than what was specified at the center and top-left? I feel it could only be different at the right and bottom edge of the region/slide where we can support different ways of handling it.

I have the behavior outlined in the following pseudocode in mind:

# calculate stride
x_stride = tile_size['width'] - tile_overlap['x']
y_stride = tile_size['height'] - tile_overlap['y']

# specify padding mode for tiles at edges
pad_mode = 'constant'  # see doc of numpy.pad for other allowable options
pad_args = {'constant_values': (0, 0)}

# pad_mode = 'reflect'
# pad_args = {}

# tile iteration
for left in np.arange(0, region_width, x_stride):
    for top in np.arange(0, region_height, y_stride):

        right = min(left + tile_size['width'] - 1, region_width)
        bottom = min(top + tile_size['height'] - 1, region_height)

        im_tile_crop = getRegion(left, right, top, bottom)  # uses tilesource.getRegion

        x_pad = tile_size['width'] - im_tile_crop.shape[1]
        y_pad = tile_size['height'] - im_tile_crop.shape[0]

        im_tile = np.pad(im_tile_crop, [y_pad, x_pad], mode=pad_mode, **pad_args)

See different options for pad_mode in the doc of numpy.pad

@cooperlab Is the behavior in above pseudocode ok? I feel this is how most sliding window analysis works.

[cooperlab]

If I was writing algorithms I would specify overlap and then calculate everything else from there. So for example I would set tile size first, then overlap, and then stride is determined from those. If segmenting objects then the coordinates would have to be put into the global frame using offsets from the stride or tile parameters (which could be returned as well). I'm open to being convinced otherwise.

[cdeepakroy]

@cooperlab -- @manthey has a PR #129 that allows us to iterate inside a region/image with a custom tile size and he also added an option to specify a tile overlap.

Regarding the overlap, his code currently crops on all edges/borders (left, top, right, and bottom) of the image/region.

Instead --- as outlined in the pseudocode i posted in my previous comment above -- i feel cropping (plus padding if necessary) should be done only for the tiles at the right-most and bottom-most border when the tile extends outside the region/image. This is more conventional from a computer vision standpoint.

What do you prefer?

Either choice we make, i feel most computer vision algorithms don't give much importance to the data at the border of an image. So it is just a matter of convention.

[cooperlab]

I agree with cropping at the right and bottom. Basically you want to have uniform tile sizes everywhere except for tiles along these edges. A simple interface would have a binary switch to pad these edge tiles to the uniform size using a color value that defaults to white/background.

[manthey]

Added in PR #129.