Add FastVLM #41112
Merged
Add FastVLM #41112
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
Contributor
|
@kamila-chay is this ready for review? |
Contributor
Author
|
@jackzhxng sorry for the delay, I had trouble with GPU access, today/tomorrow I should be able to wrap everything up and it will be ready :) |
kamila-chay
force-pushed
the
add_FastVLM
branch
3 times, most recently
from
57adb18 to
15ceb09
Compare
Contributor
Author
|
Hi @zucchini-nlp, we can start the review process, everything is ready except for the tests. I'm writing them now but I think I can do that in the meantime while responding to any comments from your end:) |
Member
zucchini-nlp
left a comment
zucchini-nlp
left a comment
There was a problem hiding this comment.
Thanks @kamila-chay , looks really nice and clean. Let's standardize a bit and then we can request core maintainer's review
docs/source/en/model_doc/fast_vlm.md
Outdated
Comment on lines
216
to
224
| ## Resources | ||
|
|
||
| A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with image-to-text transformers (here using Llava as an example). | ||
|
|
||
| <PipelineTag pipeline="image-to-text"/> | ||
|
|
||
| - A [Google Colab demo](https://colab.research.google.com/drive/1qsl6cd2c8gGtEW1xV5io7S8NHh-Cp1TV?usp=sharing) on how to run Llava on a free-tier Google colab instance leveraging 4-bit inference. | ||
| - A [similar notebook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVa/Inference_with_LLaVa_for_multimodal_generation.ipynb) showcasing batched inference. 🌎 | ||
|
|
Member
There was a problem hiding this comment.
not needed unless it is directly related to FastVLM
Comment on lines
218
to
231
| # only this value makes sense in FastVLM (we can't have a CLS token in conv layers) | ||
| if vision_feature_select_strategy != "full": | ||
| raise ValueError( | ||
| f"Unexpected select feature strategy: {vision_feature_select_strategy}, Only 'full' is supported in FastVLM." | ||
| ) | ||
|
|
||
| if any( | ||
| layer >= 0 | ||
| for layer in ( | ||
| vision_feature_layer if isinstance(vision_feature_layer, Iterable) else [vision_feature_layer] | ||
| ) | ||
| ): | ||
| raise ValueError(f"Only negative vision feature layer values are supported. Got {vision_feature_layer}") | ||
|
|
Member
There was a problem hiding this comment.
let's delete this. Having in config init is enough for now
Comment on lines
307
to
311
| >>> prompt = "<|im_start|>user\n<image>\nWhat's the content of the image?<|im_end|>\n<|im_start|>assistant\n" | ||
| >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" | ||
| >>> image = Image.open(requests.get(url, stream=True).raw) | ||
| >>> inputs = processor(images=image, text=prompt, return_tensors="pt").to(device) |
Member
There was a problem hiding this comment.
lets use chat template for code snippet
kamila-chay
force-pushed
the
add_FastVLM
branch
2 times, most recently
from
ed83be0 to
ef4bce6
Compare
Contributor
Author
|
@ArthurZucker @Cyrilvallez it's been over a month, pinging you in case you forgot 😊 |
Collaborator
|
I did I am sorry |
ArthurZucker
approved these changes
Nov 21, 2025
Collaborator
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
LGTM perfect 🤗 Thanks @zucchini-nlp for the review!
Collaborator
|
run-slow: auto, fast_vlm |
Contributor
|
This comment contains models: ["models/auto", "models/fast_vlm"] |
Contributor
CI ResultsModel CI Report❌ Failed tests
|
Member
|
@kamila-chay we can merge when the CI turns green, The weight tying failures aren't related to the model specifically, I will rebase and see if it was fixed on main already. There was a huge refactor recently and the main branch is a bit unstable upd: We need to run |
Member
|
@bot /style |
Contributor
|
Style fix runs successfully without any file modified. |
Member
|
oh wow, style bot doesn't run fix copies, didn't know about that 🥲 |
Contributor
Author
|
Okk let me look at this issue and fix it, i've been away for a few days and didn't see |
Member
|
A simple fix-copies might fix it all :) |
Member
|
run-slow: fast_vlm |
Contributor
|
This comment contains models: ["models/fast_vlm"] |
Contributor
CI ResultsModel CI Report❌ Failed tests
|
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, fast_vlm |
kamila-chay
force-pushed
the
add_FastVLM
branch
from
c53f1c5 to
e73894c
Compare
kamila-chay
force-pushed
the
add_FastVLM
branch
from
e73894c to
8e8c12a
Compare
Contributor
Author
|
Everything's green now, slow tests should be ok too :) @zucchini-nlp |
Member
|
run-slow: fast_vlm |
Contributor
|
This comment contains models: ["models/fast_vlm"] |
Contributor
CI Results✅ No failing test specific to this PR 🎉 ! |
Member
|
Thanks, great work @kamila-chay ! Let's merge 🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR adds FastVLM from Apple. The model's architecture is very similar to LlaVA, the main difference is that it uses a very fast hybrid encoder called FastViTHD. Timm's FastViT implementation is used and the LlaVA modality connector has been slightly modified.
Addresses (#38765)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@zucchini-nlp @ariG23498