Widget object detection

[mishig25]

Implemented object detection widget.

Request to Inference API (same as input for Image Classification widget):

const requestBody = { file }; // file is an image file

Response from Inference API:

Array<{ 
    label: string; 
    score: number;
    box: {xmin: number; ymin: number; xmax: number; ymax: number;} 
}>

Sample output:

[
  {"score": 0.9982, "label": "remote", "box": {"xmin": 40, "ymin": 70, "xmax": 175, "ymax": 117}},
  {"score": 0.9955, "label": "couch", "box": {"xmin": 0, "ymin": 1, "xmax": 639, "ymax": 473}},
  {"score": 0.9987, "label": "cat", "box": {"xmin": 345, "ymin": 23, "xmax": 640, "ymax": 368}}
]

Screenshots:

initial state	success state	fail state

Check out the interactive netlify demo

cc: @Narsil @NielsRogge @osanseviero please feel free to make any UX suggestions

[beurkinger]

The image can overflow on mobile resolutions

[beurkinger]

On mobile, when I click on a selection box it both select the box AND open the file selection screen.

Some ideas for a work around:

• when an image have been uploaded to the dropzone, hide the "dropzone" component and display a "remove" button or icon to reset the state
• deactivate mouse clicks on the uploaded image through css (pointer-events: none; if memory serves me well), except for the selection boxes

[beurkinger]

see comments :)

[mishig25]

@beurkinger

1. fixed the mobile overflow issue
2. regarding On mobile, when I click on a selection box it both select the box AND open the file selection screen.
   This only happens if you are switching to mobile chrome dev tools after the page is loaded (i.e mobile is being checked onMount). That is to say, this issue won't happen on an actual mobile phone. Try changing to chrome mobile dev tools and reload the page (the problem won't happen) widgetdropzone gets excluded on mobile devices here

[beurkinger]

Two things that bother me a little bit :

• lack of consistency between the two "Browse for image" inputs on mobile (see screenshot below)
• the HTML rendered by the backend for the Object Detection input is different than the one applied by the FE, and you can see the transition between the two

[beurkinger]

Almost there ! The current CSS solution work great for desktop and mobile, BUT doesn't work on tablets.

That being said, I have the solution: I just remembered that I had added a tailwind screen breakpoint util (or whatever it is called) to the moon-landing tailwind.config.js file, while trying to resolve exactly the same sort of issue:

screens: {
	'with-hover': { raw: '(hover: hover)' },
	'no-hover': { raw: '(hover: none)' },
},

So if you have the right tailwind config, you should to do stuff like with-hover:absolute instead of using size breakpoints !

[beurkinger]

See comment, let's discuss it !

[beurkinger]

Nice !