Heap Allocation Bypass [BPE]

[eamonsippy]

Heap Allocation Bypass: When train_bpe encodes a raw text string into base character IDs, it avoids standard string memory allocations entirely. What specific data structure (char_to_id) does it use to achieve $O(1)$ lookups, and why is this faster?

[Eamon2009]

The code uses a fixed-size integer array: int char_to_id[256];. Instead of instantiating C++ std::string objects and running them through a hash map (which requires slow heap memory allocation and hashing algorithms), the tokenizer simply casts a character to its raw ASCII/byte value ((unsigned char)c) and uses it as a direct $O(1)$ index to find the ID. this is when you pass a .txt file
like for eg:

.\llm.exe data\input.txt