fix t5 tokenizer and prompt token failures #1966
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| @@ -47,6 +47,8 @@ def __init__(self, model_id_or_path: str, properties: dict, **kwargs): | |||
| :param properties (dict): other properties of the model, such as decoder strategy | |||
| """ | |||
| self.lmi_dist_config = LmiDistRbProperties(**properties) | |||
| self.model_type = getattr(kwargs.get("model_config", None), | |||
| "model_type", None) | |||
There was a problem hiding this comment.
QQ: I see we are populating model_config, Where do we populate "model_type"
There was a problem hiding this comment.
model_config here is hugging face config.json which has model_type populated.
There was a problem hiding this comment.
Oh misread it, Model_type is read from model_config. Cool
| @@ -75,6 +75,9 @@ def update_request_cache_with_output(request_cache: OrderedDict, | |||
| if "prompt_tokens_details" not in request_cache[ | |||
| request_id] and request_output.prompt_logprobs: | |||
There was a problem hiding this comment.
Curious, request_output should not have prompt_logprobs as well right?
There was a problem hiding this comment.
from what I can read, request_output.prompt_logprobs is set to [float("nan")] and seems like it's passing that check.
maaquib
approved these changes
May 28, 2024
sindhuvahinis
approved these changes
May 28, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
lmi-dist, t5 model impl is different from other models and preprocessor's tokenizer attribute containstokenizerdirectly instead ofTokenizerGrouprequest_output.prompt_token_idsis an integer of bos token and fails in the for loop enumeration.