Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add SigLIP 2 #36323

Merged
merged 89 commits into from
Feb 21, 2025
Merged

Add SigLIP 2 #36323

merged 89 commits into from
Feb 21, 2025
+5,570 −122

Conversation

qubvel
Copy link
Member

@qubvel qubvel commented Feb 21, 2025 •
edited
Loading

What does this PR do?

Fixes #36318

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Sorry, something went wrong.

qubvel added 30 commits February 8, 2025 15:36
@qubvel
Docs

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
412af8c
@qubvel
Inits
15cd4b0
@qubvel
Auto classes
dea7c1b
@qubvel
Add siglip base
a3bc5e2
@qubvel
Add base tests
b39e4e8
@qubvel
Fix Siglip V1 for fix res version
87d3c51
@qubvel
Add image processor
fda45fb
@qubvel
Update conversion
2cae39f
@qubvel
Experimenting with vectorized embeddings
8f7b4ba
@qubvel
Fixup
9d9b515
@qubvel
Add modular Siglip2Processor
5ff1b57
@qubvel
Add modular configuration
bcc8282
@qubvel
Rename num patches
69a4a82
@qubvel
Correct image and text features merging
3fd6e29
@qubvel
Working conversion script
0c85c9a
@qubvel
Refactoring conversion script
5531808
@qubvel
Remove unused code in conversion script
8a73f32
@qubvel
Shorten dict a bit
d01ed76
@qubvel
Refactoring conversion
1cf65c8
@qubvel
Done conversion refactoring
fd69948
@qubvel
Fixup
1e20fb9
@qubvel
Modular siglip2
e07fa7d
@qubvel
Make model exportable and compilable without graph breaks
496e835
@qubvel
Remove position_ids from image_processor
d01e68e
@qubvel
REmove position ids from modeling file
875e627
@qubvel
Update modular
ecfa936
@qubvel
Type hint
4cbd2a0
@qubvel
Fixup
8923186
@qubvel
Set defaults to processor
869b20c
@qubvel
Add integration test
bf584bd
@qubvel qubvel marked this pull request as ready for review February 21, 2025 08:19
LysandreJik
LysandreJik approved these changes Feb 21, 2025
View reviewed changes
Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick approval as long review process happened in the private repo

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@qubvel
Fix name in toctree
Loading
Loading status checks…
b774664
@zucchini-nlp zucchini-nlp mentioned this pull request Feb 21, 2025
Closed
@qubvel
Fix test
Loading
Loading status checks…
01c0c7f
@qubvel qubvel merged commit a957b79 into huggingface:main Feb 21, 2025
23 checks passed
@EricWiener
Copy link

Hi, it looks like the vocab file references a file on a private desktop:

transformers/src/transformers/models/siglip/convert_siglip_to_hf.py

Lines 137 to 143 in a957b79

def get_vocab_file_from_model_name(model_name: str) -> str:
# get vocab file
if "i18n" in model_name:
vocab_file = "/Users/nielsrogge/Documents/SigLIP/multilingual_vocab/sentencepiece.model"
else:
vocab_file = "/Users/nielsrogge/Documents/SigLIP/english_vocab/sentencepiece.model"
return vocab_file

which may be related to #36665

@qubvel
Copy link
Member Author

qubvel commented Mar 17, 2025

Hey @EricWiener, this is just a conversion script, it's used once to convert the model from JAX implementation to HF model format and upload it to the HF Hub, and conversion script not used somehow for further model inference

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LysandreJik LysandreJik

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Siglip2 support
4 participants
@qubvel @HuggingFaceDocBuilderDev @EricWiener @LysandreJik