Skip to content

🍯 TokenJar v1.1.1: AI Token Optimizer — Windows Hardening & 100x Faster Indexing

Choose a tag to compare

View all tags
@github-actions github-actions released this 06 Oct 10:24
· 5 commits to main since this release
v1.1.1
fb675db

🍯 TokenJar v1.1.1: AI Token Optimizer — Windows Hardening & 100x Faster Indexing

This release brings critical stability hardening for Windows environments, protects LLM context windows against tool output interception by IDE agents, and delivers a 100x speedup in AST symbol indexing.

🐛 Bug Fixes & Windows Runtime Hardening

  • Windows Quote Preservation (cmd.exe /C): Replaced default Windows argument escaping with raw_arg("/C ...") in Rust and preserved command-line argument quotes in tokenjar run, eliminating SyntaxError: unterminated string literal on nested quotes.
  • Console UTF-8 Injection: Injected PYTHONIOENCODING=utf-8 and PYTHONUTF8=1 into child process execution environments, preventing UnicodeEncodeError crashes on Windows cp1254/OEM code pages.
  • Context Interception Protection: Lowered smart reader auto-paging ceiling to 40 lines and added a 2,500-byte accumulation guard (MAX_OUTPUT_BYTES = 2500), preventing IDE frameworks (Gemini / Antigravity / Windsurf) from intercepting tool payloads into external output.txt files.
  • Directory Tree Guardrails: Expanded directory skip list with data, logs, dist, build, .venv, .cache, added a 15-item per-directory truncation cap (... +N more items), and a 60-line global circuit breaker to get_directory_tree_tool.
  • Daemon/Watcher Non-Blocking Steering: Enhanced run_command_smart schema and agent steering guidelines to mandate background=true for continuous servers, bots, and file watchers, with responsive timeout diagnostic messages.
  • Windows Self-Update Safeguard: Implemented timestamped backup rename (tokenjar-{timestamp}.old) on Windows self-update to prevent os error 5 (Access is denied) when previous .old files are write-locked.

⚡ 100x Faster SQLite Indexing & Cross-Platform Path Normalization

  • Batch Metadata Reading: Introduced PersistentCache::get_all_file_metas to batch-read file hashes and modification timestamps before directory traversal, reducing 500 individual SQLite connection open/close queries down to a single query and $O(1)$ in-memory lookups (<0.5ms).
  • Raw Byte-Level Blast Radius Filtering: Added fast byte substring containment in find_symbol_references before full string allocation and AST decoding, saving hundreds of megabytes of heap allocations on large codebases.
  • Cross-Platform Path Normalization: Implemented normalize_project_root across all PersistentCache operations to strip Windows \\?\ UNC prefixes and normalize backslashes (\\ -> /), ensuring 100% cache hits across SQLite databases on Windows, Linux, and macOS.

📦 Installation & Upgrade

Using TokenJar CLI (Self-Update):

tokenjar update

Using Cargo:

cargo install tokenjar --locked

Using Pip:

pip install -U tokenjar

Pre-built Binaries:
Download the matching archive below for your platform (Windows x64, Linux x64, macOS Apple Silicon / Intel), extract, and add to your PATH.

Full Changelog: v1.1.0...v1.1.1

Full Changelog: v1.1.0...v1.1.1

Assets 9
Loading