-
Notifications
You must be signed in to change notification settings - Fork 29k
special_image_mask handling can get hit by accidental same embedding value at certain dims #38012
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
@lancercat It should not be an issue if embeds are obtained using the same model with the same |
No. |
Ah, oke, so IIUC the issue, when doing prefix tuning some virtual inputs embeds get assigned an image token idx. I still dont think this needs a fix, because comparing the diff to an arbitrary If you are doing prefix tuning with PEFT, we can try to fix it on PEFT side (though I think PEFT doesn't expand embeds but rather cache). If it was a custom script for tuning, I suggest to initialize virtual embeddings to be non-equal to the image token |
I am already meddling with the PEFT. Anyway, if the embedding comparison behaviour is not intended, maybe remove the logic? |
The feature works when users pass |
Uh oh!
There was an error while loading. Please reload this page.
transformers/src/transformers/models/gemma3/modeling_gemma3.py
Line 1262 in 5c47d08
FWIS should be testing if the [whole vector] is same enough to the special token embedding,
instead of testing same at each dim,
which should be something like
The text was updated successfully, but these errors were encountered: