Skip to content

Micro-Bundle Tokenization & Cache API Integration

Pre-release
Pre-release

Choose a tag to compare

@qwatts-dev qwatts-dev released this 17 Feb 04:51
f40b2e7

This release drastically reduces the project's dependency footprint while maintaining full end-to-end interactive inference capabilities, thanks to a direct architecture suggestion from the creator of transformers.js, @xenova.

Key Updates:

  • Micro-Bundle Tokenization: Replaced the full @huggingface/transformers library with the newly released, standalone @huggingface/tokenizers package. This shrinks the required dependency bundle from ~1.2MB down to a microscopic ~8.3kB.
  • Browser Cache API: Implemented a custom fetchWithCache wrapper utilizing the native browser Cache API (caches.open). This ensures the Llama 3 tokenizer.json and config files are only downloaded once from the Hub and are instantly loaded from local storage on subsequent runs.
  • Syntax Updates: Refactored the text-to-tensor mock bridge to utilize the updated tokenizer.encode() syntax required by the standalone module.
  • Verified Determinism: Confirmed that the new, ultra-lightweight tokenization pipeline still feeds perfect, deterministic seeds into the WebGPU engine, maintaining bit-exact output across A16, M3, and M2 Max architectures.