bert_tokenizer 是 google bert 模型tokenizer的rust实现。同时提供了对python/c/c++的支持
python 版可以通过
pip install bert_tokenizer
安装，win/linux/macosx均提供支持
对c和c++应用，引用ffi/bert_tokenizer.h头文件，并链接
ffi/bert_tokenizer/libbert_tokenizer.{so,a,dylib,dll}.
-
Notifications
You must be signed in to change notification settings - Fork 1
mklf/bert_tokenizer
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
About
No description, website, or topics provided.
Resources
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published