Make local embeddings reliable: truncate over-budget inputs and surface silent fallbacks

[zzet]

Summary

Makes provider: local survive a real repository. Today the zero-dependency local embedding path (pure-Go Hugot, MiniLM-L6-v2) dies on the first code chunk longer than 512 tokens — which every real repo has — and every failure along the chain is silent: the vector index is thrown away wholesale, the daemon logs a provider name that isn't what's running, and an embedding: block in the global config file disappears without a warning. The net effect (per #232) is that only static (GloVe-50d) and api work, with no way to see why.

This fixes the crash, isolates the remaining failure modes, and makes every layer of the chain tell the truth about what got constructed, what fell back, what was dropped, and why.

Fixes #232.

Root cause

The pure-Go Hugot tokenizer path never applies max_position_embeddings, so any input past the model's context window reaches inference at full length and produces a tensor shape mismatch. The indexer's abort-on-any-error contract then throws away the entire vector index, dropping the daemon to text-only search — silently. ("Emits empty vectors" in the issue was a misdiagnosis; short texts embed fine.)

What changed (one commit per fix)

• Token-exact truncation (internal/embedding/truncate.go) — reads the model's tokenizer.json/config.json, derives the budget from max_position_embeddings − 2, and cuts over-long inputs on a token (and rune) boundary before inference. A rune-count fast path skips the extra tokenizer pass for inputs that already fit; a degraded tokenizer falls back to a rune clamp with a warning instead of disabling the backend. Wired into both the Hugot and GoMLX providers.
• Provider output validation — every EmbedBatch now checks it returned one vector per input, none empty, each of the expected width; a violation is a loud, index-naming error instead of a silently empty/mis-dimensioned index.
• Indexer vector-ingest hardening — malformed vectors are dropped and counted (with a dropped= field and a sample of node IDs); an all-invalid build aborts to text-only rather than shipping an empty vector index.
• Truthful selection + logging — a SelectionReport records which local backend was constructed and which were skipped; the "embeddings enabled" line now reads local (hugot/fp32), dim: 384 or local → static fallback, dim: 50 instead of echoing the configured name. A genuine degradation to static warns per backend; benign misses (the onnx/gomlx stubs in a default build) stay at debug.
• Global embedding: config — GlobalConfig learns an embedding: block (merged under the repo-local one, repo values win), and unrecognised top-level keys in ~/.gortex/config.yaml now warn at startup instead of vanishing.
• GoMLX build tags — the gomlx entry points move to -tags "embeddings_gomlx XLA" so the real XLA session is actually compiled (it was dead code before); CI compiles both the pair and embeddings_gomlx alone. Docs are honest that XLA runtime is experimental and the pure-Go backend is the reliable default.
• eval embedders observability — records the concrete vector-build failure (LastVectorBuildError) and reports it instead of a bare "no vector data", with a WARN-level logger so the indexer's warnings are visible.
• Docs — ONNX manual-setup honesty, truncation semantics, config placement/precedence, a per-build-tag backend matrix, and a troubleshooting section keyed on the actual log lines.

Verification

• Unit tests for every fix (truncation span-cutting, budget derivation, output validation, the drop/all-invalid indexer paths, selection descriptions, the config merge + unknown-key detection, the eval note selection).
• Live regression (GORTEX_TEST_LIVE_MODELS=1): a ~1000-token input — the exact case that used to crash — embeds to a clean 384-dim vector through the real cached MiniLM model, as does a mixed over-budget/short batch.
• go test -race green on the touched packages; golangci-lint clean; the default, embeddings_gomlx XLA, and embeddings_onnx builds all compile with no system XLA/ONNX libraries (both are runtime dlopen).

The issue's three failure modes now each produce either working 384-dim vectors or a loud, documented, correctly-attributed log line — no silent static fallback remains reachable from a local config.