Widget for text to image generation

[osanseviero]

Input would be text
The output would be the image

[mishig25]

More than happy to take this one.
@osanseviero @julien-c please let me know if there's anything I need to improve in terms of the process I have taken for huggingface/hub-docs#14 and huggingface/hub-docs#15
Otherwise, I'll follow similar approach and implement other widgets

[osanseviero]

You're going full speed! ⚡

The approach has been great so far, I think you can go ahead with the widget.

[mishig25]

Awesome! 🤗

[mishig25]

Created draft PR #131

[osanseviero]

Some example motivation for this :)
https://discuss.huggingface.co/t/generate-gif-reply-to-english-text-with-vqgan-clip/7166

[patil-suraj]

Some example motivation for this :)
https://discuss.huggingface.co/t/generate-gif-reply-to-english-text-with-vqgan-clip/7166

And this as well huggingface/transformers#12281 😉

[borisdayma]

Hi, I'm interested in this widget.
What's the progress?

[osanseviero]

There is an existing PR - #131 but at the moment this is not integrated to the Inference API (since there is no transformers pipeline for this nor in the other libraries). If you want to build a demo for this, consider using Spaces instead for the time being.

[osanseviero]

Btw, if you have model repos with a text to image generation model, I would be interested if you share them + some example on how to do inference.

[borisdayma]

Would it make sense to allow custom pipelines?
We're working on a mini DALL-E repo so we'll need something completely custom

[osanseviero]

Yes, having a generic docker image to support custom inference is in our roadmap. We will require a very specific input/output requirement for the API to work, but other than that it will be up to the users to do their own implementations, including adding their custom dependencies.

FYI @Narsil @julien-c

[julien-c]

They can already submit a PR against https://github.com/huggingface/huggingface_hub/tree/main/api-inference-community though no?

[osanseviero]

@julien-c you mean in the form of a new docker image just for this use case?

[julien-c]

Yes – the boundary between single model and "library" being rather slim as we've seen in other cases, wouldn't mini-dall-e be a good image name in api-inference-community?
The image can also always support different checkpoints down the line. WDYT?

[mishig25]

Should we change the API output: from image to an array of images?

That way, we can show multiple generated images from a single text prompt, as it's currently done in most papers/demos/tweets. (See attached an image from @borisdayma's tweet)

If the output changes to an array of images, I'm thinking that the widget would involve some kind of horizontal scrolling/switching between different images

[osanseviero]

Yes – the boundary between single model and "library" being rather slim as we've seen in other cases, wouldn't mini-dall-e be a good image name in api-inference-community?

If this is a short-time solution, let's do it! @borisdayma, feel free to open a PR about this.

For one-shot cases, instead of having to maintain a very large number of images, I would prefer to go down the generic image path so we don't end up with hundreds of images. I think the us from the future will thank us going that road. I can work on adding the generic docker image the first week of August (in two weeks from now).

Should we change the API output: from image to an array of images?

That way, we can show multiple generated images from a single text prompt, as it's currently done in most papers/demos/tweets. (See attached an image from @borisdayma's tweet)

I'll ask @Narsil to share his opinion on this, from an endpoint point of view it makes more sense to only return one imo. Alternatively users could maybe specify number of generated images as a param, but maybe it's just better to make multiple calls to the API

[borisdayma]

I'm not sure if the best here would be to work with the inference API or HF spaces (which probably has more flexibility for these cases).

[julien-c]

I'll ask @Narsil to share his opinion on this, from an endpoint point of view it makes more sense to only return one imo

Personally I'd like widgets to be a 1:1 representation of the underlying API call, so if we want to display multiple generated images in one click, I'd be in strong favor of doing it at the API level

(for text generation, we do have an API param to generate multiple continuations, no?)

[borisdayma]

It depends on what you wanna do exactly.
For example for our demo we generate a bunch of images and then display only the best ones (8 over 64 or 128).

[julien-c]

Yep in that case Spaces are more flexible. (whereas widgets are meant to be the canonical 1:1 representation of a model's interface)

[mishig25]

(can confirm: for text generation, num_return_sequences param exists for multiple continuations)

[osanseviero]

Closing this issue since now there's a widget!