Recommendation: Multiple segmentation runs for each frame

[MartinKlevs]

Calculating multiple slightly shifted segmentation masks would let us make a higher resolution combined mask. The segmentation input would be shifted by a few pixels up or down each time it is executed (usually in a square grid). Next the masks are upscaled and shifted by the same amount and then averaged. This is a form of image super resolution. This all could be done in parallel.

This same technique can also be extended to include seperate larger jumps sideways to increase the span of the mask. Each larger jump would then be followed by the small super resolution related jumps.

The idea to cover the whole image was previously mentioned by @BenBE in #58 (comment)

This can also be extended to slightly change the sampling positions between frames, and then average two or more frames.

[BenBE]

We have basically three fields where we could apply techniques to allow for multiple passes:

1. Multiple passes for same resolution, but slightly shifted (improved resolution of mask)
2. Multiple passes for same resolution, but with reduced overlap to cover a large area
3. Multiple passes for different resolutions to enhance where to look

Let's go by the third topic first: This would basically downsample the original input image until it fits into the net completely. With that detection mask you'd use a bigger resolution to enhance on the resolution in the areas where the mask tells you about detections. Priorities could be ordered in a way that branches of that detection tree with previous detections from a previous run are favoured when evaluating the next section to analyze. Thus you could potentially process the whole image in full detail, but shortcut at any point where you don't have any more regions where previously stuff was detected. Every now and then you'd keep processing running until all areas are completed and if there's any new area popping up you could use this to update your order for processing, thus including newly found sections.

The second is my suggestion from my comment in #58: Sweep over the whole image in such a way that you cover the whole image, BUT also ensure there's some overlap (~30-50%) between sections left to allow for situations when using only non-overlapping areas would cause a mal-detection.

The first point from above finally could be used to enhance the resolution of the final mask.

All in all this would require the segmentation step to be parallelized and to allow for "save points" in processing that could be cancelled from further processing. Additionally we'd probably need some sort of task queueing for this?

[MartinKlevs]

Yes. Scaling could also be great, especially with high res inputs. This should be something that scales with the input resolution. To me this all sounds like something we can do with affine transforms.
Multiple samples can also help eliminate false positives.
Purely from a practical standpoint, I think spanning the whole image should be the first thing that is worth implementing. The borders can be a bit jarring.