Skip to content

Adding support for GraniteDoclingHybrid#44445

Draft
gabe-l-hart wants to merge 23 commits intohuggingface:mainfrom
gabe-l-hart:GraniteDoclingHybrid
Draft

Adding support for GraniteDoclingHybrid#44445
gabe-l-hart wants to merge 23 commits intohuggingface:mainfrom
gabe-l-hart:GraniteDoclingHybrid

Conversation

@gabe-l-hart
Copy link
Copy Markdown
Contributor

@gabe-l-hart gabe-l-hart commented Mar 4, 2026

What does this PR do?

This PR adds support for the forthcoming Granite Docling model based on the Granite 4 LLM architecture (GraniteMoeHybrid).

Draft Status

This PR is in draft pending the possibility of some additional changes:

  • Finalizing the vision projector
  • Finalizing the name as GraniteDoclingHybrid (versus eg GraniteMoeHybridDocling or similar)

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Models:

zucchini-nlp
zucchini-nlp reviewed Mar 5, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Left a few comments so we are aligned with recent changes in v5, otherwise the PR is in good shape

src/transformers/models/granite_docling_hybrid/processing_granite_docling_hybrid.py Outdated
**kwargs,
)

image_seq_len = image_seq_len if image_seq_len is not None else self.image_seq_len
Copy link
Copy Markdown
Member

@zucchini-nlp zucchini-nlp Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

am I right that this line is the only diff from Idefics3? If yes, I'd prefer to just copy completely from ideifcs because making image_seq_len an arg isn't very useful

Copy link
Copy Markdown
Contributor Author

@gabe-l-hart gabe-l-hart Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real diff is below where the GotOcr2 logic is swapped for the Idefics3 logic. I think this may be one of the TBD architectural decisions, so if the team decides to switch back to Idefics3, this ould all go away.

@gabe-l-hart gabe-l-hart force-pushed the GraniteDoclingHybrid branch 2 times, most recently from f2b4745 to a16c63c Compare March 9, 2026 21:11
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Mar 10, 2026

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gabe-l-hart
feat: Initial implementation of GraniteDoclingHybrid
c399691 
This is untested, but passes code generation correctly now

Branch: GraniteDoclingHybrid
AI-usage: draft (IBM Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
Co-authored-by: omenetti.matteo@gmail.com
Co-authored-by: nassarofficial@gmail.com
@gabe-l-hart
feat: Register granite_docling_hybrid in all auto registries
20a6df0 
Branch: GraniteDoclingHybrid
AI-usage: full (IBM Bob)
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Add missing config forwarding for underlying text model
2c6094b 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Fix how image_hidden_states are extracted from get_image_feature…
11d8217 
…s after v5 rebase

Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Update processing kwargs to not have return_row_col_info
c471581 
This was getting injected in Idefics3ProcessorKwargs and causing errors
since it's not a valid kwarg in the base ImageProcessorKwargs and somehow
that's what was being validated against (versus
Idefics3ImageProcessorKwargs)

Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
chore: regen modeling
f4b8f26 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
style: Linting fixes
f166660 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: No inline imports
43b3c7b 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Remove unnecessary entry in MODEL_FOR_PRETRAINING_MAPPING_NAMES
9629b3c 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
style: Fix copyright headers year
4c596a6 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Remove dead/unused code/docstrings from review
d4d327a 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
feat: Remove return_dict plumbing
00ce088 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
feat: Consolidate config + processing into modular
1848d83 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
chore: regen from modular after consolidation
4b73a3e 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Remove cache initialization logic in forward
1e5f7bd 
This all lives in `prepare_inputs_for_generation`

Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Remove unnecessary class-attrs for GraniteDoclingHybridProcessor
1945128 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Use auto_docstring for GraniteDoclingHybridProcessor
c99c53c 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
feat: Use modern image fetching
5122f17 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
chore: redo codegen
2d72415 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Remove image_seq_len as a kwarg
b6d26d6 
This was recommended in review. Also, there was already a bug where
self.image_seq_len was being used for the real computation, so this kwarg
was never fully supported.

Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
chore: Redo codegen
1ed97ab 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
fix: Correctly update for no more cache_position
bee7b27 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart
chore: regen from modular
f82599a 
Branch: GraniteDoclingHybrid
AI-usage: none
Signed-off-by: Gabe Goodhart <ghart@us.ibm.com>
@gabe-l-hart gabe-l-hart force-pushed the GraniteDoclingHybrid branch from 1bf67fb to f82599a Compare March 11, 2026 19:18
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 11, 2026

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zucchini-nlp zucchini-nlp zucchini-nlp left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gabe-l-hart @HuggingFaceDocBuilderDev @zucchini-nlp