Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Support for Florence-2 model family #21118

Open
theolivenbaum opened this issue Jun 20, 2024 · 1 comment
Open

[Feature Request] Support for Florence-2 model family #21118

theolivenbaum opened this issue Jun 20, 2024 · 1 comment
Labels
feature request request for unsupported feature or enhancement model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc.

Comments

@theolivenbaum
Copy link

Describe the feature request

As per https://huggingface.co/microsoft/Florence-2-large-ft/discussions/7, it seems like the model type is not yet supported by the converter:

Can we get an Onnx version of this model for use on Windows .NET using the onnxruntime?

I tried to convert it but it failed.
optimum-cli export onnx --trust-remote-code --model microsoft/Florence-2-large-ft ./models/Florence-2-large-ft_onnx/

ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Florence-2-large-ft.f3c2bbf1d042a8976e0c43b3a3ead3f53a8dad88.configuration_florence2.Florence2Config'> for this kind of AutoModel: AutoModelForVision2Seq.
Model type should be one of BlipConfig, Blip2Config, GitConfig, Idefics2Config, InstructBlipConfig, Kosmos2Config, LlavaConfig, LlavaNextConfig, PaliGemmaConfig, Pix2StructConfig, VideoLlavaConfig, VipLlavaConfig, VisionEncoderDecoderConfig.

Describe scenario use case

New Florence-2 model family should be supported by onnxruntime

@theolivenbaum theolivenbaum added the feature request request for unsupported feature or enhancement label Jun 20, 2024
@github-actions github-actions bot added model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc. platform:windows issues related to the Windows platform labels Jun 20, 2024
@theolivenbaum theolivenbaum changed the title [Feature Request] Support for Florence-2 family [Feature Request] Support for Florence-2 model family Jun 20, 2024
@sophies927 sophies927 removed the platform:windows issues related to the Windows platform label Jun 20, 2024
@shubham0204
Copy link

@theolivenbaum Xenova from HuggingFace has uploaded the ONNX models for Florence-2 on onnx-community repo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request request for unsupported feature or enhancement model:transformer issues related to a transformer model: BERT, GPT2, Hugging Face, Longformer, T5, etc.
Projects
None yet
Development

No branches or pull requests

3 participants