Fix models like phi-3 which have a mismatched tokenizer definition and model output tensor size #792
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Build failure seems to be unrelated to this change @riedgar-ms |
|
Doesn't this need to be combined with the Phi3-enabling PR? |
Yes, oops, meant to merge that in (assuming you mean the one that adds phi-3 to the tests). Just did. |
riedgar-ms
approved these changes
May 3, 2024
|
Build failures on Mac seem to be due to build machine size issues, not this change. I can run all tests on my Mac. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Phi-3 Mini models (e.g. https://huggingface.co/microsoft/Phi-3-mini-4k-instruct) have a total tokenizer vocab size of 32011 (base 32000 plus 11 special tokens), but the model outputs logit vectors of size 32064 (for CUDA performance reasons maybe?).
I think models that use this trick will always have the "padded" model output at the end, which makes this PR a very clean and simple change to guarantee tokenizer <> model output consistency. This fix will NOT work if the padded outputs are either in the beginning or interspersed throughout the token vector, but I haven't seen a model do that yet. If models do start doing that, we'll need to make significantly heavier changes across the guidance codebase, as we currently heavily rely on the len(tokenizer) to be usable for iteration.