Add Bottleneck T5 (e.g., thesephist/contra-bottleneck-t5-small-wikipedia)

[keviddles]

Name of the feature
I'd love to leverage https://huggingface.co/thesephist/contra-bottleneck-t5-small-wikipedia in the browser

It is not listed in the supported models above, though I have been able to convert it locally.

However, I can't import it in JS as I get t5 is not supported.

I guess my question is, is this lack of support because it can't be supported, or simply because it hasn't been implemented yet?

If the latter, I'd be happy to hack on this myself if it's something you think could be supported, though I'm not quite sure where to start. Is there a guide for contributors to contribute community models?

(Let me know if it's helpful to share the converted models, they're about 800mb)

Reason for request

I'd love to port this colab to Javascript: https://colab.research.google.com/drive/1CF5Lr1bxoAFC_IPX5I0azu4X8UDz_zp-?usp=sharing#scrollTo=1hwxFKZzFKn5

[xenova]

Hi there 👋 the t5 model type has been supported for quite a while, so you shouldn't be getting an error like that. Have you uploaded your models to the Hub already? If so, can you share the model ids?

When it comes to weird issues like that, it's usually because of outdated build tools (like create-react-app, or something). Could you share what tools and versions you are using?

[keviddles]

Ah, interesting. I'll put a repo together and share it shortly

[keviddles]

I uploaded a repository here (https://github.com/keviddles/contra-bottleneck-t5-demonstration), but I'm having trouble getting it to deploy to github pages. Will comment if I can get that working.

I think the issue is probably that I'm loading the model wrong. The code I'm using to load the model is:

  const modelPath = "thesephist/contra-bottleneck-t5-small-wikipedia";
  const tokenizer = await AutoTokenizer.from_pretrained(modelPath);
  const model = await AutoModelForCausalLM.from_pretrained(
    "thesephist/contra-bottleneck-t5-small-wikipedia"
  );

And that produces the error:

Uncaught (in promise) Error: Unsupported model type: t5
    at AutoModelForCausalLM.from_pretrained (models.js:3630:19)
    at async main (index.ts:14:17)

Alternatively, if I load the model with:

  const model = await T5PreTrainedModel.from_pretrained(
    "thesephist/contra-bottleneck-t5-small-wikipedia"
  );

I see:

I started with the former command (using AutoModelForCausalLM) as I'm trying to translate from the original Python code here.

[keviddles]

Have you uploaded your models to the Hub already? If so, can you share the model ids?

Oh, follow up question: must models be available on the Hub, or can they be available locally? I've only been trying locally, I didn't realize that they might need to be uploaded to the hub first.

[xenova]

I uploaded a repository here (https://github.com/keviddles/contra-bottleneck-t5-demonstration), but I'm having trouble getting it to deploy to github pages. Will comment if I can get that working.

Thanks! At first glance, your usage looks correct. Could you just let me know which transformers.js version you have installed? We did previously have some issues with minification (which lead to a similar error message as the one you are seeing). Fortunately, those have been resolved now in 2.6.2

Oh, follow up question: must models be available on the Hub, or can they be available locally? I've only been trying locally, I didn't realize that they might need to be uploaded to the hub first.

No, you can use the models locally too 👍 I just wanted to test, so if you have them on the hub, it will be easier to see what the problem is :)

I assume you've already converted it to ONNX, right? (note that transformers.js can't load the model you have listed out-of-the-box, since it doesn't contain the ONNX weights + graph). See here for more information.

[keviddles]

Could you just let me know which transformers.js version you have installed?

^2.6.2

No, you can use the models locally too 👍 I just wanted to test, so if you have them on the hub, it will be easier to see what the problem is :)

Gotcha, let me upload them to the hub so they're available for testing.

I assume you've already converted it to ONNX, right? (note that transformers.js can't load the model you have listed out-of-the-box, since it doesn't contain the ONNX weights + graph). See here for more information.

Yep, converted them and they seemed to convert fine.

[xenova]

I think I know what the problem is: t5-models aren't generally used as causal LMs (decoder-only / text-generation)... Instead, they are seq2seq LMs (encoder-decoder / text2text-generation). However, the model you are working with is indeed a text-generation model, since they use custom modelling code to handle this.

self.model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True).to(self.device)

Also, optimum doesn't actually support exporting t5 models for text-generation (see here). Instead, it only supports text2text-generation. I would assume that it will export dummy encoder weights when you exported with optimum (even though you will only be using the decoder during inference). cc @fxmarty any ideas?

[keviddles]

Ah, interesting feedback. So presumably, the exported model is not viable.

I uploaded the models here, if there's some easy way to tell whether it's viable or not: https://huggingface.co/keviddles/contra-bottleneck-t5-demonstration-onnx/tree/main

[xenova]

You may be able to export it to ONNX with custom configurations, a recent addition to the Optimum library. Also, since the "Bottleneck T5" architecture is custom code (built on top of HF transformers), it may be a good idea to make a feature request in the original transformers library too.

[keviddles]

Thanks for the feedback, @xenova .

It'll probably be a while since I pick this back up, so I will close this issue for now. If others want to pick it up, feel free to reopen.