Smart Annotation

[bbernhard]

I am thinking about an API endpoint which takes an image uuid and a image mask as input. This data then gets fed into a microservice running the grabcut algorithm and returns some data points.

Proof of Concept

The user draws a rectangle around the region of interest. Furthermore it's possible to draw scribbles in order to mark foreground and background pixels which again improves the algorithm.

Whenever the user draws something on the canvas, a mask gets created which is then sent to the API endpoint. For the above example, the mask would look like this:

Challenges

• Peformance: grabcut isn't the fastest algorithm, so it's quite challenging to make that work in near realtime. Can we improve the performance by downscaling the images before applying grabcut? Can we use densecut[1] instead of grabcut?

[1] http://mmcheng.net/densecut/

[bbernhard]

Another idea:

Lazy Snapping [1] (might be even better when it comes to realtime performance)

[1] https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/lazysnapping_siggraph04.pdf

[dobkeratops]

I guess running this on the server as suggested is going to be more consistent than the client (such a range of devices with varying cpu/gpu performance/versions) .. so I wont rush into wasm/webgl ideas for that just yet..

[bbernhard]

Personally I would see the server solution more like a intermediate solution..I really would like to see that running in the browser some day. Mainly because the feature requires a lot of development and maintainence efforts on the server side (which I want to get rid of some day, if possible):

• quite slow, as we always talk to the server (so it's really hard to make that work in "realtime")
• needs server resources, proper monitoring and the possibility to scale horizontally (in case there are a lot of requests)
• needs a active internet connection (not a big disadvantage yet; but maybe we can support offline annotation some day).

I do not have any benchmarks, but I could imagine that modern (mobile) devices are already pretty powerful...so I would really be interested to see a browser based approach. If you have planned to do something similar which runs in the browser, please don't drop the idea just because of that :)

Before I settled with the server side solution I tried to find something which runs in the browser, but couldn't find something that "just works" (all the projects I found were either abandoned or written in Javascript, which makes them pretty slow). So I think there is definitely a place for such a thing!