Check if the given token is a string

[silverriver]

Some model use bytes as their tokens, such as Qwen (see: https://huggingface.co/Qwen/Qwen-7B/blob/ef3c5c9c57b252f3149c1408daf4d649ec8b6c85/tokenization_qwen.py#L136 )

[rlouf]

Thank you! Is this solving an issue you encountered? Also would you mind running pre-commit to fix code style?

[silverriver]

Thank you! Is this solving an issue you encountered? Also would you mind running pre-commit to fix code style?

Yeap, this solves my issue when I am using the Qwen model family, such as https://huggingface.co/Qwen/Qwen-1_8B.

Here is a code snip:

import vllm
from outlines.serve.vllm import RegexLogitsProcessor
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen-1_8B', trust_remote_code=True)
prompt = "Answer with A, B, C, or D."

llm = vllm.LLM(model="Qwen/Qwen-1_8B", trust_remote_code=True)
logits_processor = RegexLogitsProcessor(regex_string=r'[ABCD]', llm=llm)
outputs = llm.generate([prompt],
            sampling_params=vllm.SamplingParams(
                max_tokens=5, logits_processors=[logits_processor]
            ),
        )

I have also found out that the Qwen model family use a non-standard tokenizer implementation, i.e., they return byte sequences as their tokens, not strings. However, I think it is always benifical to consider more corner cases.