You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Using pad_token, but it is not set yet.
Traceback (most recent call last):
File "inference-test.py", line 135, in <module>
outputs = pipe(inputs,
File "/data0/pratyush/DeepSpeedExamples/inference/huggingface/text-generation/utils.py", line 69, in __call__
outputs = self.generate_outputs(input_list, num_tokens=num_tokens, do_sample=do_sample)
File "/data0/pratyush/DeepSpeedExamples/inference/huggingface/text-generation/utils.py", line 107, in generate_outputs
input_tokens = self.tokenizer.batch_encode_plus(inputs, return_tensors="pt", padding=True)
File "/data0/pratyush/miniconda3/envs/ds/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2829, in batch_encode_plus
padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
File "/data0/pratyush/miniconda3/envs/ds/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2466, in _get_padding_truncation_strategies
raise ValueError(
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
I tried to set a pad token / replace the tokenizer in the following three ways:
Yet, 1 and 3 still result in the same error as above. 2 yields:
ValueError: The following `model_kwargs` are not used by the model: ['token_type_ids'] (note: typos in the generate arguments will also show up in this list)
Expected behavior
Updated tokenizer should work, at least?
A couple of other related questions:
Is there a list of supported models for DeepSpeed Inference w/ meta tensors?
I read that DeepSpeed-Inference supports automatic tensor parallelism for Huggingface models by default, but I need meta_tensors to load larger models. However, meta_tensors seem to require kernel injection (at least for OPT models, which error out otherwise), so I've been using that. Is my understanding correct / is that the right approach?
Describe the bug
Galactica model running text-generation does not have a pad token.
git clone https://github.com/microsoft/DeepSpeedExamples.git cd DeepSpeedExamples/inference/huggingface/text-generation deepspeed --num_gpus 8 inference-test.py --name facebook/galactica-6.7b --ds_inference --use_meta_tensor --use_kernel
I tried to set a pad token / replace the tokenizer in the following three ways:
pipe.tokenizer.pad_token = pipe.tokenizer.eos_token
pipe.tokenizer.add_special_tokens({'pad_token': '[PAD]'})
pipe.tokenizer = AutoTokenizer.from_pretrained("facebook/galactica-6.7b")
Yet, 1 and 3 still result in the same error as above. 2 yields:
Expected behavior
Updated tokenizer should work, at least?
A couple of other related questions:
Rest of the setup is exactly the same as in #3871
Thanks!
The text was updated successfully, but these errors were encountered: