Widget Image Segmentation

[mishig25]

Implemented image segmentation widget.

Request to Inference API (same as input for Image Classification widget):

const requestBody = { file }; // file is an image file

Response from Inference API:

Array<{ 
    label: string; 
    score: number;
    mask: string 
}>

Sample output:

[
    {"score": 0.9982, "label": "remote", "mask": "iVHIJKLMNOPQRSTUVWXYZabcd..."},
    {"score": 0.9955, "label": "couch", "mask": "iVHIJKLMNOPQRSTUVWXYZabcd..."},
    {"score": 0.9987, "label": "cat", "mask": "iVHIJKLMNOPQRSTUVWXYZabcd..."}
]

Screenshots:

success state	success state (mask selected)

initial state	error state

Check out the interactive netlify demo

cc: @Narsil @osanseviero @NielsRogge

Update 1: demo should be working on Safari as well

[julien-c]

This looks super cool!

(but the link to netlify is just regular object detection for now, no?)

[mishig25]

@julien-c try the demo now 👍 https://615a24ebbb69fc346548d87c--huggingface-widgets.netlify.app/

[julien-c]

WOW 😮

[osanseviero]

Very excited about this!!! It looks too good

[gary149]

Awesome 😎. Found an issue on iOS: the browse image button works it opens my images library then I select an image but nothing happens.
Edit: I just saw your comment on safari so I guess it's expected

[Narsil]

Backend fixed !

Outdated

[beurkinger]

I get an error message while trying to run your branch:

> huggingface-widgets@0.0.1 dev /Users/thibault/Documents/web/moon-landing/server/huggingface_hub/widgets
> svelte-kit dev

 > html:/Users/thibault/Documents/web/moon-landing/server/huggingface_hub/widgets/src/routes/index.svelte:4:13: error: Expected "from" but found "{"
    4 │ ... type { Model...
      ╵    

I think but we should put lang="ts" in every <script /> tag, as it's the recommend way of proceeding by the Svelte/SvelteKit documentation. Actually I think we had more or loess decided to do it with @Pierrci, but never got around to doing it ?

Note: this error appeared after I ran npm i, which updated svelte-kit. It may well be that the next version of SvelteKit isn't compatible with the parser option that allows us to get away with not putting lang="ts" in script tags - I think I remember reading that it was deprecated (?).

[mishig25]

@beurkinger please let me know if you have any other comments on this PR.

In terms of:

huggingface-widgets@0.0.1 dev /Users/thibault/Documents/web/moon-landing/server/huggingface_hub/widgets
svelte-kit dev

we should probably take care of it in different PR

[mishig25]

Will merge it after I merge huggingface/transformers#14062 & Inference API is updated