) (#49) * fix: eliminate ghost wikilinks in LLM-generated wiki content (closes #47) `openkb lint` was flagging 100+ broken [[wikilinks]] on every fresh ingest. Investigation showed the issue is fully deterministic: - `_compile_concepts` plan stage caps "create" at 2-3 concepts/doc - but `_CONCEPT_PAGE_USER` and `_SUMMARY_USER` told the LLM to "include wikilinks to related concepts" with no whitelist - so concept/summary bodies ended up with 5-10 wikilinks each, most pointing to concepts the plan never created → ghost links Fixes in three layers: 1. Prompt whitelist (Layer A): `_CONCEPT_PAGE_USER` and `_CONCEPT_UPDATE_USER` now receive `{known_targets}` (existing concepts on disk + plan.create + plan.update + plan.related + summary for this doc). The prompt instructs the LLM to use [[wikilinks]] only for targets in the list and write plain text otherwise. 2. In-memory deferred write + summary rewrite (Layer B): `compile_short_doc` no longer writes the v1 summary to disk immediately; it's held in memory and used as cache context. After all concept bodies are generated, every body is run through `strip_ghost_wikilinks` (canonicalizes via NFKC + lowercase + `_↔-` normalization, drops unresolved links to plain text). Then the LLM regenerates the summary with the same cache-friendly message structure (BP1 + BP2 hit) but constrained to the whitelist, and the v2 summary is also stripped before being written. 3. lint --fix (Layer C): `fix_broken_links` applies the same strip_ghost_wikilinks against existing wiki content so users with already-broken KBs can clean them up in place. End-to-end validation: - Same 5 documents (PDF + xlsx + docx + 2 markdown), same model (gpt-5.4-mini): old code produced 115 broken wikilinks across 30 unique ghost targets; new code produces 0 broken across 197 valid wikilinks. - Cache-control optimization preserved: summary-rewrite call hits 92% cached tokens (e.g. 11008/11887 input cached on the PDF doc). - 251 tests pass, including 19 new unit tests for `strip_ghost_wikilinks` and `_normalize_target`. Compatible with Karpathy's spec — the gist intentionally leaves the exact wikilink topology unspecified, only requiring that cross-references exist and stay consistent. * fix: harden _compile_concepts against summary rewrite failure modes Follow-up to the ghost-wikilinks fix. Self-review on the prior commit surfaced four resilience gaps in the new summary-rewrite path: 1. Plan-parse-fail / empty-plan fallback paths wrote the raw v1 summary to disk without stripping. The exact LLM failure modes most likely to produce ghost-heavy v1 summaries (malformed plan JSON, empty plan) bypassed the whole Layer B defense. 2. If the rewrite LLM call returned an empty string, `_write_summary` was called with "" and silently wiped the summary. The v1 was no longer on disk (deferred write), so there was no fallback. 3. The rewrite call had no try/except. Any transient API failure propagated out, leaving no summary file and no concept files written despite all the LLM work being done. 4. `max_tokens=2048` could truncate long summary rewrites mid-sentence while the original v1 summary call had no cap. Fixes: - Extract `_write_v1_summary_stripped()` closure inside `_compile_concepts` that strips the v1 summary against `_list_existing_wiki_targets(wiki_dir)` before writing — used by both early-return paths. - Rewrite the summary-rewrite block to a try/except + empty-check pattern: on exception or empty result, fall back to the v1 summary stripped against the full whitelist. The summary is always written. - Remove the `max_tokens=2048` cap (matches the v1 summary call). - New `TestCompileShortDocFallbacks` class covers all three fallback scenarios. `test_full_pipeline` now provides a third mock response for the summary-rewrite call and asserts the rewrite content lands on disk (the existing assertion would pass with garbage content because `_mock_completion` clamps to the last response). All 255 tests pass. * fix: strip ghost wikilinks from saved query answers and chat transcripts PR #49's compile-time defenses don't cover two other code paths that write LLM-generated content into the wiki, both flagged by self-review: - `openkb query --save` writes the agent's answer to wiki/explorations/ (cli.py). The query agent's instructions (via schema_md) encourage [[wikilinks]] but the agent's view of which pages exist can drift from disk reality. - `/save` in the chat REPL writes the session transcript with the same exposure (chat.py). Both files land in wiki/explorations/, which lint.py scans for broken links. So ghost wikilinks emitted by the agent end up reported as broken links on the next `openkb lint` run, even after PR #49. Promote `_list_existing_wiki_targets` to `lint.list_existing_wiki_targets` (public, paired with `strip_ghost_wikilinks`) so it can be reused outside the compile pipeline. The two save paths now strip the agent output against the on-disk target set before writing. User-typed turns in the chat transcript are preserved verbatim — only assistant responses are filtered, since intentional user input shouldn't be auto-rewritten. Tests: - `TestQuerySaveGhostStrip` exercises `openkb query --save` with mixed valid + ghost wikilinks in the mocked answer. - `test_save_transcript_strips_ghost_wikilinks` covers the chat path, also asserting user turn content is preserved. - 257 tests pass. * perf+refactor: cache known_targets via 3rd breakpoint, reuse norm_index, fix update prompt Three follow-ups from the perf/design self-review. None of them gated correctness but they sharpen cost and clarity: 1. **Cache the wikilink whitelist (BP3).** Previously every concept generation call and the summary-rewrite call injected the full `{known_targets}` list (5–10k tokens for a 500-concept KB) into the per-call user prompt — past BP2 — so the whitelist was billed on every request. Hoist it into a new cached user message (`known_targets_msg`) sitting between `summary_msg` and the per-call user turn, marked with `cache_control: ephemeral`. Plan call deliberately omits it (whitelist isn't known yet at plan time, and plan uses `concept_briefs` instead). Cache structure is now: BP1 = doc, BP2 = summary, BP3 = whitelist. Concept and rewrite calls reuse all three; only the trailing per-concept (or per-rewrite) prompt is fresh. The new `_KNOWN_TARGETS_USER` prompt lives in one place; the `_CONCEPT_PAGE_USER` / `_CONCEPT_UPDATE_USER` / `_SUMMARY_REWRITE_USER` templates lost their `{known_targets}` slot and now reference "the whitelist message above". Test coverage updated to verify the BP3 marker on both concept and rewrite calls. 2. **Reuse the normalized target index across loops.** `strip_ghost_wikilinks` used to rebuild `norm_index = {_normalize_target(t): t for t in known_targets}` on every call. `fix_broken_links` (lint --fix) scanned N files with the same `known_targets`, paying the O(M) rebuild N times — measurable on large KBs. Same shape for `_save_transcript` on a long chat. Expose `build_norm_index(known_targets)` as a small public helper, add a keyword-only `norm_index` parameter to `strip_ghost_wikilinks`, and pre-build the index once in `fix_broken_links` and `_save_transcript`. Behavior is identical when the parameter is omitted, so all existing callers are unchanged. 3. **Resolve the `_CONCEPT_UPDATE_USER` instruction contradiction.** The update prompt said "Maintain existing [[wikilinks]] and add new ones where appropriate" immediately followed by "MUST link only to targets in this whitelist". For pages with legacy ghost links, this was a contradictory directive — the strip safety net caught the resulting waste but the prompt itself was unclear. Reword to make the whitelist the unconditional rule, with "preserve structure and intent" as the orthogonal instruction. 260 tests pass (3 new: prebuilt-norm_index identity test, build_norm_index unit tests). The existing cache-control test is extended to assert BP3 on both concept-generation and summary-rewrite calls.

A lot of broken links were found. #47

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions