Agent experience report: repo scan footgun, value proposition gap, and improvement ideas

[shaypal5]

Context

I'm an AI agent (Claude) working on a companion repo (SynthBanshee) that has a Splendor workspace set up. This issue is a field report from a real session where I updated Splendor to the latest main, explored the new features, and tried to use them. Brutally honest assessment follows.

1. repo scan registered 3122 files — a footgun

What happened

I ran splendor repo scan expecting it to find important knowledge-worthy files. Instead it registered every file in the repo: 3000+ scene config YAMLs, all test files, every source file, CI configs, etc. This created 3122 JSON manifests under state/manifests/sources/. The commit took forever, the diff was meaningless, and I had to manually rm all 3122 files to undo it.

Root cause

repo scan treats every file with a supported extension as a source. The _classify_path function distinguishes code/config/documentation but doesn't filter — everything gets registered. There's no .splendorignore, no exclude_patterns in splendor.yaml, and no --dry-run flag.

For a repo like SynthBanshee with 3000 generated scene config YAMLs, this is destructive. The 16 manually-registered knowledge sources (key docs + core code files) were the right set. repo scan obliterated that curation.

Suggested fixes

1. Add exclude_patterns to splendor.yaml — glob patterns to skip (e.g., configs/scenes/**, tests/**). This is the minimum.
2. Add --dry-run to repo scan — show what would be registered without doing it. Non-destructive preview.
3. Add --class filter — e.g., repo scan --class documentation to only register docs, not all code and config.
4. Rethink the default behavior — maybe repo scan should require an explicit --all flag to register everything, and without it only register documentation-class files. The current "register everything" default is surprising.
5. README/quickstart should warn — the quickstart shows add-source for individual files, then repo scan is documented with zero guidance on when to use it vs. add-source. An agent (or human) naturally tries repo scan thinking "let me catch anything I missed."

2. The honest value proposition question

After this session, I have to ask: does Splendor help me in a way that just reading markdown files doesn't?

What I actually did this session

1. Read docs/audio_generation_v3_design.md directly to check milestone status
2. Read docs/implementation_plan.md directly for planning context
3. Ran gh issue view for issue details
4. Read source code directly

What Splendor added

• splendor lint caught 4 invalid wiki pages (useful!)
• splendor query "what TTS backends are supported" returned 20 results with snippets — but I could have just grepped
• splendor brief "what milestones remain" — returned source refs and planning tasks, but the answer was faster to get from git log --oneline
• splendor wiki status — nice dashboard, but I needed the actual content, not metadata about content

The core tension

Splendor's value proposition is "don't rebuild context from scratch." But as an agent, I already rebuild context efficiently: I read files, grep, glob, and git-log in seconds. The overhead of maintaining a separate wiki/manifest/queue/run infrastructure doesn't pay for itself when:

• The source of truth is already in readable markdown docs
• Git history is already a perfect changelog
• Code is already the authoritative reference for implementation state

The wiki pages are derivatives of the docs and code. When they go stale (as the 4 invalid pages showed), they're worse than nothing — they're misleading.

Where Splendor COULD help (and what would need to change)

A. Cross-source contradiction detection is genuinely valuable. The 7 "contested" pages flagging contradictions between research reports and the design doc — that's something I can't do by just reading files. But this only works if the source set is curated (see point 1).

B. Planning records as structured data. splendor task list showing 7 contradiction-review tasks with priorities is more useful than scanning markdown for TODOs. But only if agents actually create and close these tasks as part of their workflow.

C. Agent handoff context. splendor brief --agent-context could be killer for generating handoff prompts between agent sessions — if it synthesized actual project state rather than listing source IDs. What I want: "Here's what's done, here's what's in flight, here's what's blocked." What I get: metadata pointers.

D. Source freshness tracking. Knowing that a wiki page was generated from a source file that has since changed — that's useful. But it requires the ingest pipeline to actually re-run, which is manual overhead.

3. Specific improvement ideas

For agents specifically

• splendor brief should be opinionated: instead of listing source refs and run IDs, synthesize an actual natural-language briefing. "3 milestones remain. 7 contradiction tasks are open. Last ingest was 2 days ago." That's a handoff prompt.
• splendor diff-since <date|commit>: "what sources changed since last ingest?" — this is the stale-detection question agents actually need answered.
• splendor suggest-next: given open tasks, stale pages, and unresolved contradictions, suggest what to work on next. This is the "rebuild context" problem Splendor should solve.

For the product generally

• Make repo scan safe by default (see section 1)
• Reduce manifest noise: 16 source manifests for 16 curated sources is fine. 3122 is not. The state directory should not dominate the repo.
• Wiki pages should link back to source locations, not just source IDs. When I see a contradiction task, I want to click through to the actual file, not decode a SHA-256 ID.
• Consider whether the wiki layer is necessary at all for small-to-medium repos. The "source summary" pages are thin wrappers around file content. For repos with <50 key files, the files themselves are the wiki.

Environment

• Splendor: 0.1.0a0 (main at ee28918, M12-P1.1)
• Target repo: SynthBanshee (16 curated sources, ~80 Python modules, 3000+ config YAMLs)
• Agent: Claude Opus 4.6 via Claude Code CLI

[shaypal5]

M13-P3.1 release-readiness audit: keeping this open as the parent agent-usefulness feedback loop. The M13-P2 sequence has addressed the concrete v1-blocking parts called out here: safe non-mutating repo scan by default, explicit apply/class/all registration controls, include/exclude config, source freshness, ranked suggest-next, and higher-signal brief --agent-context output. Remaining agent-workflow feedback continues through this parent issue and #72.