Replace custom HF tokenizer with C++ bindings for HF tokenizer #153
Description
Refer to these C++ bindings for HF tokenizer by mlc-ai: https://github.com/mlc-ai/tokenizers-cpp/
Metadata
Metadata
Assignees
Labels
No labels
{{ message }}
Refer to these C++ bindings for HF tokenizer by mlc-ai: https://github.com/mlc-ai/tokenizers-cpp/