Description
We need to introduce a Byte-Pair Encoding (BPE) tokenizer infrastructure in tokenizer.h for the llm.cpp . Currently, the header lacks a comprehensive sub-word handling mechanism, which causes raw training data and chat inputs to fall back to crude representations or risk out-of-vocabulary errors.
This issue tracks the integration of vocabulary mapping, encoding/decoding pipelines, and binary/text file serialization to persist merge ranks across runtime sessions.
Tasks
BPE Data Structures:
-
Implement vocabulary map structures (vocab) and reverse lookup tables for token-to-string and string-to-token conversions.
-
Store and manage merge priority ranks (merge_ranks) loaded from training artifacts.
Encoding Pipeline (encode):
-
Convert raw input text strings into sequences of integer token IDs.
-
Implement iterative sub-word merging based on adjacent byte-pair priority ranks.
-
Add safe fallback encodings for raw character streams to prevent crashes on unseen characters.
Decoding Pipeline (decode):
- Reconstruct human-readable text output streams from arrays of token IDs.
Serialization & File I/O:
-
Implement disk-loading and saving mechanisms for vocabulary weights and merge rules.
-
Add comprehensive error handling for missing, malformed, or corrupted tokenizer files.
Expected Behavior
-
Raw text inputs passed to dl.encode() must correctly resolve to valid token IDs based on the learned BPE vocabulary.
-
Token arrays passed to dl.decode() must accurately reconstruct the original text without truncation or artifact corruption.
-
The system should gracefully catch missing tokenizer files or invalid paths, outputting informative error diagnostics instead of aborting abruptly.
Environment
OS: Cross-platform (Linux / macOS / Windows)
Compiler / Toolchain: GCC / Clang / MSVC supporting C++17 or higher
Branch / Version: master
#69 fix
Description
We need to introduce a Byte-Pair Encoding (BPE) tokenizer infrastructure in tokenizer.h for the llm.cpp . Currently, the header lacks a comprehensive sub-word handling mechanism, which causes raw training data and chat inputs to fall back to crude representations or risk out-of-vocabulary errors.
This issue tracks the integration of vocabulary mapping, encoding/decoding pipelines, and binary/text file serialization to persist merge ranks across runtime sessions.
Tasks
BPE Data Structures:
Implement vocabulary map structures (vocab) and reverse lookup tables for token-to-string and string-to-token conversions.
Store and manage merge priority ranks (merge_ranks) loaded from training artifacts.
Encoding Pipeline (encode):
Convert raw input text strings into sequences of integer token IDs.
Implement iterative sub-word merging based on adjacent byte-pair priority ranks.
Add safe fallback encodings for raw character streams to prevent crashes on unseen characters.
Decoding Pipeline (decode):
Serialization & File I/O:
Implement disk-loading and saving mechanisms for vocabulary weights and merge rules.
Add comprehensive error handling for missing, malformed, or corrupted tokenizer files.
Expected Behavior
Raw text inputs passed to dl.encode() must correctly resolve to valid token IDs based on the learned BPE vocabulary.
Token arrays passed to dl.decode() must accurately reconstruct the original text without truncation or artifact corruption.
The system should gracefully catch missing tokenizer files or invalid paths, outputting informative error diagnostics instead of aborting abruptly.
Environment
OS: Cross-platform (Linux / macOS / Windows)
Compiler / Toolchain: GCC / Clang / MSVC supporting C++17 or higher
Branch / Version: master
#69 fix