Widget for image captioning

[osanseviero]

Would have as input

• Image

Would output

• Text

This should be similar in implementation to the image classification widget.

[mishig25]

I've implement ImageCaptioningWidget here.

input/request: identical to ImageClassificationWidget
const requestBody = { file }; // uploaded image

output/response: identical to TextGenerationWidget
[ { "generated_text": "..." } ]

Questions:

1. I've assumed ImageCaptioningWidget api contract (request and response specifications) from similar widgets. Please let me know if there's a better way to find api contracts for the widgets
2. What modelId should I supply here?
3. Would there be icon for ImageCaptioingWidget?
4. Is there a preferred way to mock/stub the api response?

[osanseviero]

That's very cool! Feel free to open a draft PR so we can discuss over code 😄

1. Your proposed API contract seems fine to me. I'll send a PR to add this pipeline (FYI @Narsil in case you have ideas)
2. At the moment we have no model for this task. You can remove it for the time being :)
3. Yes, we should have an icon. @julien-c any ideas?

[Narsil]

1. LGTM too.

2. There are not "spec" like openAPI or something like it, aside from code right now.

[mishig25]

@osanseviero removed ModelData from widgets/src/routes/index.svelte and opened a draft PR huggingface/huggingface_hub#115

[julien-c]

Would there be icon for ImageCaptioingWidget?

Feel free to propose one @mishig25!

[julien-c]

@mishig25 our icon style is from https://icones.js.org/collection/carbon (cc @gary149)

[julien-c]

@osanseviero

At the moment we have no model for this task. You can remove it for the time being :)

Let's plan to add/build one otherwise this widget won't be terribly useful :)

[osanseviero]

Let's plan to add/build one otherwise this widget won't be terribly useful :)

Yeah, I'm on it right now :)

huggingface/huggingface_hub#114 added the pipeline to the API validation, now we need to add it to one of the docker images (I was thinking to a new Keras image).

[mishig25]

@julien-c how does this one look: carbon:align-box-bottom-center ?

[julien-c]

@mishig25 yeah looks good to me

[mishig25]

@julien-c added the icon and pushed the changes

[osanseviero]

This is now done for text-to-image