Skip to content

v0.7.0

Choose a tag to compare

@Zelys-DFKH Zelys-DFKH released this 21 May 01:47
· 53 commits to main since this release

Minor release: new context-saving features (Grep output compression, a session orientation brief, bash loop detection) plus a round of worker, database, and compaction fixes.

Added

  • Grep output compression. Large grep/rg/ag/ack results (>30 lines) are compressed to a file-level summary: top 20 files by match count, totals included, full output cached for token-goat bash-output recall. Typical savings: ~80%.

  • Bash loop-detection escalation. The same command run twice triggers a "ran 2×" escalation; three or more repeats produce a "WARNING: ran N×" advisory. Stops runaway loops from burning context unnoticed.

  • Session-wide hint deduplication. Identical hints are suppressed after their first injection within a session. SHA-256 fingerprinting with a JSON-persisted hints_seen set means the agent never gets nagged twice for the same file.

  • Session orientation brief. At session start in a dirty git repository, a compact block (~50 tokens) is injected: current branch, modified/staged/untracked counts, and the five most-recent commits. Disable via TOKEN_GOAT_SESSION_BRIEF=0 or [session_brief] enabled = false in config.toml.

  • Adaptive PreCompact manifest budget. The manifest budget scales from 200 to 600 tokens based on edit count, symbol accesses, and bash activity. Sessions with little activity get a lean manifest; complex ones get the full picture.

  • Git diff --stat in PreCompact manifest. A git diff --stat HEAD summary (capped at 8 lines / 200 chars) is now included in the compaction manifest. The compaction LLM always sees which files drifted from the last commit, even when the session cache doesn't list them as edited.

  • Symbol names in re-read hints. Re-read hints now include up to three symbol names previously accessed in the flagged file (e.g., [symbols: login, get_user, Session]), so the agent can decide whether token-goat read file::symbol is sufficient.

  • Error-preserving smart truncation. When bash output exceeds the size cap, the trimmed view keeps: first 10 lines + up to 10 error-signal lines with 2-line context + last 10 lines, separated by --- N lines omitted ---. Errors are never lost to truncation.

  • Loaded version in token-goat stats. The stats report now shows the running token-goat package version: a header line in the ANSI renderer (token-goat v0.6.1), the version in the rich fallback renderer's panel title, and a top-level version field in --json output. Confirms at a glance which build produced the numbers.

Fixed

  • Git-history indexing batches its writes in one transaction. _index_history_inner inserted up to 200 commit rows on an autocommit connection (isolation_level=None), so every INSERT committed on its own and the trailing conn.commit() was a no-op: 200 separate fsyncs and 200 writer-lock acquisitions per reindex sweep. The batch now runs inside a single BEGIN/COMMIT, acquiring the lock and committing once. The last_indexed_at staleness marker is also written only when at least one commit stored, so a batch that wholly failed (for example, a database that stayed locked throughout) no longer stamps itself "indexed" and suppresses the retry for an hour.

  • project_writer_lock acquisition is now atomic. _try_acquire checked lock_path.exists() and then write_text — a check-then-write with a TOCTOU window: two callers that both observed the file absent each wrote the lock and each believed it held it, so two index_project runs could write the same per-project database concurrently. Acquisition is now a single os.open(O_CREAT | O_EXCL) create — the atomic-mutex pattern the worker slot claim already uses — and _stale falls back to the lock file's mtime so the brief create-then-write window can't be misread as a dead lock.

  • Git-history indexing moved to the background worker. The SessionStart hook spawned git_history.index_project_history on a daemon=True thread inside the hook process, which exits within milliseconds — killing the thread before the indexing finished. Git-history hints are now refreshed by the worker's periodic reindex sweep, which runs in a durable process; index_project_history is idempotent and staleness-gated (1 h), so the move adds no measurable cost.

  • Worker claim-slot no longer wedges on a write failure. If os.write failed after _try_claim_worker_slot created the claim file, the file descriptor leaked and an empty claim file was left on disk. _worker_claim_is_stale treats an empty claim as not-stale (to protect the create-then-write window), so that orphan could never be reclaimed and the single-worker slot stayed blocked. The fd is now closed and the empty file removed on a write failure. Separately, run_daemon wrapped its claim-file cleanup in a finally whose try began only after _write_pid / _register_autostart / cleanup_on_startup, so an exception in any of those skipped the cleanup — the try now covers all startup work.

  • Session-start git brief is capped by one shared deadline. _build_session_brief ran three git subprocesses (rev-parse, status, log) sequentially, each with a fixed 2 s timeout, so a slow or pathological repository could stack a ~6 s pause onto session start. The three calls now share a single ~2.5 s wall-clock budget, and a call is skipped once the budget is spent.

  • A deferred dirty-queue drain no longer slows re-indexing. On Windows a concurrent enqueue_dirty can hold dirty.txt open, making os.replace fail with a sharing violation; drain_dirty_queue retries and then defers. It returned [] for that case — indistinguishable from a genuinely empty queue — so the worker counted a deferred drain as an idle cycle and let adaptive back-off drift re-indexing toward its 10 s maximum while edits piled up. drain_dirty_queue now returns None on a deferral, and the worker resets the idle counter instead of incrementing it.

  • token-goat doctor no longer integrity-checks the production database. The stats summary opened global.db through the read-write path, which runs PRAGMA integrity_check on connect — multi-second on a large global.db, and it created the database file as a side effect when one did not exist yet. The summary now reads through open_global_readonly(), so doctor stays fast regardless of database size and never mutates the database it is diagnosing.

  • token-goat stats breakdown rows now rank by share. The "By kind", "By day", and "By project" tables emitted rows in byte-sorted order while the share column they display is token-derived, so the share percentage zig-zagged whenever bytes and tokens ranked rows differently (an image-heavy day saves bytes but ~0 tokens). Each section renderer now orders its rows by the same share metric it displays — "By source" already did this.

  • Unbounded global.db WAL growth. Every hook writes stat rows to global.db, and under a heavy multi-agent burst its passive autocheckpoints were perpetually blocked by overlapping readers, so the write-ahead-log file only ever grew — one session reached an 11 GB global.db-wal, after which every hook (including the SessionStart hook that runs on /compact) stalled for minutes scanning it. Connections now set PRAGMA journal_size_limit so the WAL file is truncated after each checkpoint, and the worker force-runs a wal_checkpoint(TRUNCATE) on global.db every maintenance cycle. A tests/test_wal_growth_guard.py regression suite, wired into the pre-commit hook, locks both halves of the fix in place.

  • Temp files and automation artifacts excluded from PreCompact manifest. Paths under /tmp/, Windows %APPDATA%, .improve-state-*.json, and improve_commit_msg_* are filtered before the manifest renders. Previously they leaked into "Files Edited" and wasted manifest budget on entries the compaction LLM couldn't use.