Draw Bounding Box in Two Clicks

[victordibia]

It would be great to support a faster annotation scheme where a bounding box is drawn by two clicks (the first is the top left and second is the bottom right).

[SkalskiP]

Hi @victordibia ,
I have to admit, I don't fully understand. At this point, you can do it with one click. Additionally, it doesn't matter if you start from the bottom to the bottom or from the bottom to the top. Why do you think that the current solution is not optimal?

[victordibia]

You are right.
With a mouse, the bounding box operation can be performed fluidly with a click, a drag (click button held down), and a release ( event complete, bounding box is created at the end of the drag)

For a user on a laptop with a trackpad, the process is slightly more complicated. The drag operation requires first a click, which has to be held down with one finger while the pointer is moved around with another finger. During this process (and as the user tries to find an accurate ending position), the hold can get dislodged and the interaction has to begin again.

A potential alternative is use two clicks - a click (top left or bottom right anchor), a mousemove (box is drawn and follows mouse movement ... no drag needed ... gives user time to find the right end anchor point without danger of needing to restart, and reduces the chances of needing adjustments) and another click (event complete, bounding box drawn.). The downside here is that there might be a slight learning curve, but in the end can be faster for users labelling alot of data. I think this paper captures more of the spirit of the above. https://arxiv.org/pdf/1708.02750.pdf

Its a minor usability feature.

P.S. I think the makesense tool is pretty incredible, fantastic work so far!

[tfriedel]

there's also this idea where instead of drawing a bounding box you click on four extreme points of the object (left, right, top, bottom). this is supposedly faster than drawing the box if you care about a tight box. here's a paper about the technique: https://arxiv.org/abs/1708.02750

[SkalskiP]

Hi, @tfriedel. You think this new/alternative labeling option makes sense?

[herruli]

Google AutoML is adopting both two points and four points bounding box. It's own labeling tool used four points.

[tfriedel]

I think it makes sense. If you really want to have a tight bounding box it's easier. Otherwise you will start in some corner and you later have to correct the box. herruli <notifications@github.com> schrieb am Do., 23. Jan. 2020, 10:26:

…

Google AutoML is adopting both two points and four points bounding box. It's own labeling tool used four points. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12?email_source=notifications&email_token=AAN3VRTFG7OUKU4Y3GTKJR3Q7FPDHA5CNFSM4IKLRB2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJWW7YI#issuecomment-577597409>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN3VRUHYXA7CUACJ3RQ6OLQ7FPDHANCNFSM4IKLRB2A> .

[bourdakos1]

I think another alternative to 4 point bounding boxes is the extended cross-hair. That way you can quickly determine 2 extreme points at once. The quickness/tightness of 4 point, but with the added speed boost of only clicking twice:

[tfriedel]

that might be even better!

[SkalskiP]

You know what's funny about it? In earlier versions of MakeSense we had an extended cross-hair. You think that would be helpful?

Plus, I saw a cool feature. What do you think about those crops that are cut out of the image and then listed in the right panel?

[SkalskiP]

@bourdakos1 and @tfriedel PR adding cross-hair is currently open for code review #90

[SkalskiP]

@bourdakos1 and @tfriedel all changes can be tested here: https://develop.makesense.ai

[tfriedel]

Hi SkalskiP, I tried the feature and it looks pretty good!
One thing I would still add is to remove the need for dragging. Just clicking in one corner and clicking again in the other should also work for creating the box.
This would make it also consistent with lines and polygons, where that is possible as well.

[johongo]

https://arxiv.org/pdf/1708.02750.pdf

[SkalskiP]

Hi, @johongo 👋! If I'm getting everything right you think it would be a good idea to enable labeling using for clicks?

[johongo]

Yessir! 😄

[SkalskiP]

Hm... I guess people would like us to support 3 forms of bounding box drawing now :)

• Click and drag
• Two clicks
• Four clicks

[johongo]

I think the argument made in the paper is pretty solid. I just thought you would find it interesting, but no stress!

[SkalskiP]

No, don't get me wrong, it is really interesting. No question about it. I'm just immediately thinking about a few things:

• How much time it will take to implement that?
• How many people would actually use that?

Judging by the number of comments on this and similar issues there is a demand in the community to implement that.

An additional thing that we need to think about if we would like to go with it is, how to document that. It may not be obvious to everyone.