Add EoMT with DINOv3 backbone#41212
Conversation
…for-eomt-dinov3-checkpoint-r4mmdc Fix attention mask dtype in EoMT-DINOv3 converter verification
…t-dinov3-model-into-transformers
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
…t-dinov3-model-into-transformers
…t-dinov3-model-into-transformers
yonigozlan
left a comment
yonigozlan
left a comment
There was a problem hiding this comment.
Hey @NielsRogge ! Looking clean but I wonder if we can simplify even further, or if we even need a new model for this (we would still need to do some refactoring to Eomt though)
| class_queries_logits=class_queries_logits, | ||
| last_hidden_state=sequence_output, | ||
| patch_offsets=patch_offsets, | ||
| ) |
There was a problem hiding this comment.
Most of this is very similar to the base Eomt, I'm wondering if we could support the dinov3 version without adding a new model? This would mean defining a backbone instead of copying submodules of dinov2/dinov3, and make position_embeddings optional in the backbone layer call, so not sure how feasible this is, also in terms of BC. Wdyt @NielsRogge ?
NielsRogge
force-pushed
the
codex/integrate-eomt-dinov3-model-into-transformers
branch
from
1f098ff to
9e24af7
Compare
…t-dinov3-model-into-transformers
|
run-slow: eomt, eomt_dinov3 |
|
This comment contains models: ["models/eomt", "models/eomt_dinov3"] |
|
run-slow: eomt, eomt_dinov3, vit |
CI Results |
|
run-slow: eomt, eomt_dinov3, vit |
|
This comment contains models: ["models/eomt", "models/eomt_dinov3", "models/vit"] |
1 similar comment
|
This comment contains models: ["models/eomt", "models/eomt_dinov3", "models/vit"] |
CI ResultsModel CI Report❌ Failed tests
|
|
@NielsRogge I've fixed the last few smaller things, can you check the failing tests at #41212 (comment) Moreso the integration tests, if that's a GPU diff (A10) lmk I can update as well in that case - sdpa dispatch one is probably since we are forced to use a mask --> suffices to skip with a comment |
|
run-slow: eomt_dinov3 |
|
This comment contains models: ["models/eomt_dinov3"] |
CI ResultsCommit Info
Model CI Report❌ 3 new failed tests from this PR 😭
|
|
@vasqu feel free to handle the integration tests, I assume this is because of the GPU difference |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, eomt, eomt_dinov3, vit |
|
run-slow: eomt, eomt_dinov3, vit |
|
This comment contains models: ["models/eomt", "models/eomt_dinov3", "models/vit"] |
|
Merging now, just fixed the last few tests due to GPU diffs resulting in slightly higher atols - updated to a10 numbers where needed (2 tests) |
5 participants
What does this PR do?
This PR adds EoMT with a DINOv3 backbone. The authors of EoMT released new checkpoints which swap the DINOv2 backbone by the newer DINOv3: https://github.com/tue-mps/eomt/blob/master/model_zoo/dinov3.md
Disclaimer
This PR was implemented using OpenAI Codex and further improved by me.
Who can review?
@yonigozlan