Bug in LLaMA fast tokenizer

[WoosukKwon]

In my environment, using the LLaMA fast tokenizer raises an error about protobuf:

  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/convert_slow_tokenizer.py", line 445, in __init__
    from .utils import sentencepiece_model_pb2 as model_pb2
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/transformers/utils/sentencepiece_model_pb2.py", line 91, in <module>
    _descriptor.EnumValueDescriptor(
  File "/opt/conda/envs/dev/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

While downgrading the protobuf version removed the error, it slowed down the initialization time by ~8x.

• Initialization with fast tokenizer & protobuf==3.20.3

real    4m18.476s
user    3m52.706s
sys     0m27.644s

• Initialization with slow tokenizer

real    0m27.620s
user    0m8.011s
sys     0m19.237s

[WoosukKwon]

@merrymercy has confirmed that this also happens in his environment. Maybe we can use the slow tokenizer until the bug gets fixed.

[zhuohan123]

+1 Met the same error on my side.

[YuboFeng2023]

Hi all,

I also encountered this problem in my environment:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/vllm/transformers_utils/tokenizer.py", line 34, in get_tokenizer
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name, *args,
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 702, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1811, in from_pretrained
    return cls._from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_base.py", line 1965, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/llama/tokenization_llama_fast.py", line 89, in __init__
    super().__init__(
  File "/usr/local/lib/python3.8/dist-packages/transformers/tokenization_utils_fast.py", line 114, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "/usr/local/lib/python3.8/dist-packages/transformers/convert_slow_tokenizer.py", line 1288, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "/usr/local/lib/python3.8/dist-packages/transformers/convert_slow_tokenizer.py", line 445, in __init__
    from .utils import sentencepiece_model_pb2 as model_pb2
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/sentencepiece_model_pb2.py", line 91, in <module>
    _descriptor.EnumValueDescriptor(
  File "/usr/local/lib/python3.8/dist-packages/google/protobuf/descriptor.py", line 796, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "demo_of_vllm.py", line 58, in <module>
    main()
  File "demo_of_vllm.py", line 53, in main
    run(argv)
  File "demo_of_vllm.py", line 41, in run
    llm = LLM(model=argv.ckpt)  # Name or path of your model
  File "/usr/local/lib/python3.8/dist-packages/vllm/entrypoints/llm.py", line 62, in __init__
    self.llm_engine = LLMEngine.from_engine_args(engine_args)
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 154, in from_engine_args
    engine = cls(*engine_configs,
  File "/usr/local/lib/python3.8/dist-packages/vllm/engine/llm_engine.py", line 80, in __init__
    self.tokenizer = get_tokenizer(
  File "/usr/local/lib/python3.8/dist-packages/vllm/transformers_utils/tokenizer.py", line 42, in get_tokenizer

But my code is very simple:

# encoding=utf-8

import os
import sys
import argparse
import logging
from vllm import LLM

__dir__ = os.path.dirname(os.path.abspath(__file__))
sys.path.append(os.path.abspath(os.path.join(__dir__, '..')))

# from Lib.TextAbstract import TextAbstract
# from Lib.Stock import Stock


def parse_argv():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--prompt",
        type=str,
        default='What is the largest animal?',
        help="text of prompt")
    parser.add_argument(
        "--ckpt",
        type=str,
        required=True,
        help="file path of checkpoint")
    parser.add_argument(
        "--log",
        type=str,
        default=None,
        help="file path of log")
    return parser.parse_args()


def run(argv):
    for k, v in argv.__dict__.items():
        logger.info("%s: %s" % (k, v))

    prompt = "Q: %s\nA:" % argv.prompt
    llm = LLM(model=argv.ckpt)  # Name or path of your model
    output = llm.generate(prompt)
    print(output)


def main():
    argv = parse_argv()
    logging.basicConfig(
        filename=argv.log,
        format='%(asctime)s - %(levelname)s - %(name)s - %(msg)s',
        datefmt='%Y-%m-%d %H:%M:%S',
        level=logging.INFO)
    run(argv)


if __name__ == '__main__':
    logger = logging.getLogger(__name__)
    main()

I run this code by following command:

python3 demo_of_vllm.py \
  --ckpt $GEMINI_DATA_IN1/open_llama_3b

where '$GEMINI_DATA_IN1/open_llama_3b' is the folder path of HuggingFace LLaMa`s checkpoint.

How to solve this problem?