Shape Error in Llama4VisionMLP2

[priyammaz]

System Info

transformers/src/transformers/models/llama4/modeling_llama4.py

Line 1096 in d1b9236

self.fc2 = nn.Linear(config.projector_output_dim, config.projector_output_dim, bias=False)

Im pretty sure this should be:

self.fc2 = nn.Linear(config.projector_input_dim, config.projector_output_dim, bias=False)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Nothing to reproduce, I just noticed a shape mistake, but i havent had a chance to run it personally yet!

Expected behavior

The output of fc1 should be the input of fc2, unless there is some hidden logic that I am missing

[priyammaz]

I now see the input and output are the same in the config, but still this can be a bug if someone changed that!

[Rocketknight1]

cc @ArthurZucker but this looks like a bug, yes!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Rocketknight1]

gentle ping @ArthurZucker - I'm pretty sure this is wrong and only works because the config variables have the same value, but I wanted to check in case the fix would break anything else!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Rocketknight1]

@priyammaz Sorry for the delay, but I think you can go ahead and make a PR for this! It seems like an obvious bug to me

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.