EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
Learning BPE embeddings by first learning a segmentation model and then training word2vec
Byte Pair Encoding (BPE)
A curated list of tokenizer libraries for blazing-fast NLP processing.
WordPiece Tokenizer for BERT models.
Detect whether the text is AI-generated by training a new tokenizer and combining it with tree classification models or by training language models on a large dataset of human & AI-generated texts.
A framework for generating subword vocabulary from a tensorflow dataset and building custom BERT tokenizer models.
A fast and lightweight implementation of a likelihood-based Byte Pair Encoding (BPE) tokeniser. Unlike traditional BPE, this tokeniser selects merge candidates based on a normalized likelihood score, improving tokenisation quality by prioritising statistically significant merges.
Fast wordpiece, sentencepiece tokenizer by Trie, OpenMP, SIMD, MemoryPool
Add a description, image, and links to the wordpiece topic page so that developers can more easily learn about it.
To associate your repository with the wordpiece topic, visit your repo's landing page and select "manage topics."