add generate to coca model

[gpucce]

This PR should add the generate method to the CoCa model to add support for generation

based on https://github.com/lucidrains/x-transformers/blob/main/x_transformers/autoregressive_wrapper.py

[gpucce]

@rom1504 where would you like to test the generative side?

I should have implemented the generation part as a generate method in the CoCa model, looks like smth is happening but not really as much as it should I believe. Perhaps the captioning loss also needs some adjustments with regards to pad tokens, I might be missing that in the coca loss.

Nevertheless, it would be nice to put up a proper benchmark, which task/data should I use for it?

[gpucce]

Also tests pass locally but it would be nice to have them run here too, I don't know if that is possible with this PR not pointing at main.

[rom1504]

Could you check with the published model (see link in other PR) that it works on a few images ?

…

On Wed, Dec 21, 2022, 16:15 Giovanni Puccetti ***@***.***> wrote: Also tests pass locally but it would be nice to have them run here too, I don't know if that is possible with this PR not pointing at main. — Reply to this email directly, view it on GitHub <#314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437R7PH3LUSVEG2XAY33WOMNIHANCNFSM6AAAAAATFTGXHU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gpucce]

@rom1504 some examples from imagenet, always uaing the prompt "the image shows a" then PRED is what the model generates, LABEL the label in imagenet and then the 5 images that are used.

PRED of a a red red - - crested crowned tur
LABEL African grey, African gray, Psittacus erithacus
PRED shows corn corn in on a a farm tractor . 
LABEL ear, spike, capitulum
PRED of a a giant giant panda panda . ( <end_of_text>
LABEL giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca
PRED shows a a wooden wooden cradle cradle with with a 
LABEL cradle
PRED of a a cowboy cowboy on sitting a on horse 
LABEL cowboy hat, ten-gallon hat

PRED shows a a red red - - backed brown cou
LABEL coucal
PRED shows a a dog dog looking and at looking a 
LABEL Italian greyhound
PRED shows the the view mountain of of the the volcano 
LABEL volcano
PRED shows a a dog dog sitting sitting on on the 
LABEL Welsh springer spaniel
PRED shows a a black black - - capped throchick
LABEL chickadee

[rom1504]

Nice!

It seems to love repeating itself
I'm wondering if we could fix that by tuning sampling params

Any opinions @lucidrains ?

[Soonhwan-Kwon]

I also tested naive caption generation and it repeats a lot. We need beam search or contrastive search to get better result and I recommend to use same convention of huggingface generation for future supports.

[Soonhwan-Kwon]

I already scored cider 120 on coco with CoCa and in this implementation seperated kv in attention as HF transformers, it can be more compatible and efficient. If needed I can have a look in here.

[rom1504]

It would definitely be great if you can have a look and make a PR!

…

On Wed, Dec 21, 2022, 19:08 Soonhwan-Kwon ***@***.***> wrote: I already scored cider 120 on coco with CoCa and in this implementation seperated kv in attention as transformers, it can be more compatible and efficient. If needed I can have a look in here. — Reply to this email directly, view it on GitHub <#314 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437QERPGFSEPGURZ5LXLWONBR7ANCNFSM6AAAAAATFTGXHU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rom1504]

which task/data should I use for it?

maybe "cider on coco", @Soonhwan-Kwon could you point to code to evaluate this ?

[rom1504]

I checked the code here and it looks good, maybe we can merge and iterate on top

[rom1504]

@gpucce could you resolve the merge conflict ? (rebase on coca)

[Soonhwan-Kwon]

which task/data should I use for it?

maybe "cider on coco", @Soonhwan-Kwon could you point to code to evaluate this ?

I used this blip code as reference. line80~
https://huggingface.co/spaces/Salesforce/BLIP/blob/fe102a54bbdf72e68c97f685d50a25ce6e46cc5e/data/utils.py

[gpucce]

Replying here instead of the closed PR @rom1504 the reason the text part is harder to rewrite using only self.text is that the CLS token for contrastive loss is a model parameter separated from token_embeddings, to avoid confusion with selecting a specific token in the vocabulary, but to add it in the right place I had to rewrite the forward, although the logic is very similar

[rom1504]

I see. I'm wondering if we should add an option to have a cls token in the original text encoder
Also, some HF encoders already have that, so in this case it would already work by default

[gpucce]

I think to support generation with shorter context length, if we add support for this cls to TextEncoder there would be need for more changes still in TextEncoder. An alternative could be defining a coca tokenizer. Whichever you think is best, I would do it in a different PR if you are still ok with merging this one

[rom1504]

Making changes in text encoder might work

I think what's important is to find a way such that we could support more text encoders kind such as HF ones for example