Skip to content

ALM base model class#45534

Draft
eustlb wants to merge 4 commits intomainfrom
alm-base-model-class
Draft

ALM base model class#45534
eustlb wants to merge 4 commits intomainfrom
alm-base-model-class

Conversation

@eustlb
Copy link
Copy Markdown
Contributor

@eustlb eustlb commented Apr 20, 2026

What does this PR do?

Fix a discrepancy in ALMs design compared to VLMs that for most of them don't have a base model class because the use the causal model for the text model directly, while Llava style (which is indeed more aligned with the lib philosophy) is rather:
XxModel: encoder + projector + language base model backbone
XxForConditionalGeneration: XxModel + lm_head

This is motivated to simplify changes required for vLLM compatibility as seen in #39330

how to ensure this is BC?

  • weight loading: thanks to the dynamic weight loader and conversion mapping
  • TODO further verifs

@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants