Micro-Bundle Tokenization & Cache API Integration
Pre-release
Pre-release
This release drastically reduces the project's dependency footprint while maintaining full end-to-end interactive inference capabilities, thanks to a direct architecture suggestion from the creator of transformers.js, @xenova.
Key Updates:
- Micro-Bundle Tokenization: Replaced the full
@huggingface/transformerslibrary with the newly released, standalone@huggingface/tokenizerspackage. This shrinks the required dependency bundle from ~1.2MB down to a microscopic ~8.3kB. - Browser Cache API: Implemented a custom
fetchWithCachewrapper utilizing the native browser Cache API (caches.open). This ensures the Llama 3tokenizer.jsonand config files are only downloaded once from the Hub and are instantly loaded from local storage on subsequent runs. - Syntax Updates: Refactored the text-to-tensor mock bridge to utilize the updated
tokenizer.encode()syntax required by the standalone module. - Verified Determinism: Confirmed that the new, ultra-lightweight tokenization pipeline still feeds perfect, deterministic seeds into the WebGPU engine, maintaining bit-exact output across A16, M3, and M2 Max architectures.