Repository navigation
Replies: 6 comments
|
Update: validated the C3 (extraction/quality) layer against a real Kiro session archive — and the data reshaped the design. I ran a curation pass over 51 real Kiro sessions (a historical backup from Jun–Jul + live sessions from this month, same host, deduplicated by 1. Session triage is a cheap O(n) gate that removes ~78% before any LLM. With a calibratable value score (threshold 0.25), 11 of 51 sessions were retained; the other 40 were tests, session-deaths, empties, and one 2. Value must be scored, not decided by a binary verdict. I made this a continuous 0–1 score with a threshold, mirroring 3. Two independent axes, not one. My first heuristic ("short session = discardable") threw away real short conversations. The fix: separate is-it-a-test? (mechanical prompt signature) from does-it-have-content? (a substantive assistant reply). A short real conversation with a reply is kept; a test ("hello", "turn N ok") or a question whose session died before any reply is dropped. 4. Language is NOT a value signal. I briefly tried penalizing prompts in a language other than the operator's locale. It worked by accident on my corpus (I happened to test in English), but it does not generalize — it would wreck an English-speaking operator's archive. Removed it: the intent signature + reply density separate tests from conversations on their own, language-agnostic. Language belongs only in the extractor (honoring the operator locale), never in value triage. 5. Discovery gap, measured. All of this is captured in the fork RFC (now v0.3, §8). It doesn't change the three-layer shape — it fills in the C3 layer with rules that came from real data. Still asking the same two questions: does the layered shape work, and is Kiro-to-YAML (now including triage) the right first increment? |
|
First concrete increment from this design is up: #1423 makes the Phase 0 harvest coverage measurable (instrumented in #1350 but with no reader). That's the groundwork for the per-agent, pluggable ingestion this discussion proposes — you can't tune per-agent extraction rules without seeing per-kind coverage first. One caveat that reinforces the per-agent framing: the only client we have validated end-to-end is Kiro, so the coverage/volume numbers we can report today are Kiro's; every other client needs its own measurement. Full compat/kill-switch answers are in the #1346 RFC thread. |
|
Status update + I'm going to start building, so this doesn't stall waiting on a reply. Since the last update I validated the per-agent/pluggable design against our own codebase and it held up — but it also surfaced a useful reality: a prototype branch I have (fact-extraction, gap-detection, passive feedback) has the core logic + unit tests done, but every piece is unwired — no scheduler job calls the pipelines, the tools aren't in the registry, and the low-score hook that would record a "gap" isn't called from the search path. Lots of tested logic, zero integration. That reframes the work from "write features" to "wire what exists + reconcile." So the plan I'll follow, smallest-first, each behind an opt-in env flag and fork-only until it proves out:
I'll keep each as a focused PR when it earns its place (measured, not asserted). None of this blocks on you — I'm flagging it so the design discussion and the implementation stay in sync. If any of the layering here contradicts how you'd want the pluggable ingestion to look, now's the cheap moment to say so. |
|
Sorry this sat for eight days while you kept measuring. Short answer to your two questions: yes to the layered shape, and yes to Kiro-to-YAML as the first increment, but only in its original form from 30.09. The 05.10 version grew, and I'd like to pull it back apart. 1. The first increment is two PRs, not one. The 30.09 plan was "extract Kiro's rules into So:
Note that 2. C2 stays data. Globs, kind-to-role maps, injected-marker lists, cutoffs, format detection: that belongs in YAML, the same way 3. C1 identity uses The author identity from RFC #1100 already exists: 4. C3 lives in #1346. I'm not re-gating the extractor here. #1346 is the canonical thread for it, and I still owe you a reply there on your 02.10 answers to points 2 and 3. That reply comes separately, so the decisions stay in one place. 5. The orphan pipelines are out of scope for this discussion. Fact-extraction and gap-detection are new subsystems with new tools and scheduler jobs. None of it is on 6. Pin the RFC. Same as on #1345: please put the fork RFC version you want me to review in the discussion body, pinned by commit. It's been moving (v0.3 in your update), and I want to review a fixed text. Next step is yours: open PR A (YAML extraction only, golden on candidates and coverage). Everything else waits behind it. |
|
RFC pinned for review + reference map (your point 6). Reference map (so the thread doesn't drift):
Pinned RFC (fixed text to review): This is v0.3, the umbrella RFC for the harvest ingestion layers. It absorbs the three prior RFCs as layers — source-identity (C1), kiro-sessions (C2), design-extraction (C3) — and §8 carries the calibratable value score + language-agnostic triage from the 05.10 update. One honesty note: this commit predates your reply above, so its status line still reads "awaiting Henry" and doesn't yet reference #1481. I kept it unmodified on purpose — it's the exact text that was on the table when you reviewed. The next revision will fold in your two-PR split and re-pin here with a new commit, so the version under review never moves silently. Next step on my side: PR A (YAML extraction only, golden on candidates + coverage), targeting #1481. |
Uh oh!
There was an error while loading. Please reload this page.
This is a design proposal, not a PR — I'd rather align on the shape before writing code, and the execution comes in small measured increments (first one below).
The problem, in one line: harvest quality depends on the importer understanding each agent's format and semantics, and today it has neither the per-agent rules nor the multi-source model to do that well. Three symptoms that turned out to be one problem:
harvest/parser.py, ~10 of 12 agent-specific rules are data disguised as code (kind→role maps, injected-marker lists, cutoffs, format detection, globs). Adding an agent means editing the parser in 6+ places, risking regression in the three already supported.MCP_HARVEST_SESSION_DIR). In practice one memory service serves N clients — Kiro CLI, Kiro workspace/IDE, Kiro Crew (SQLite), and potentially OpenClaw / OpenCode / Hermes — each with its own path, format, rule profile, and identity.The proposal: a declarative, pluggable, multi-source ingestion layer, in three layers:
{agent, path, profile, identity}. Declarative with assisted auto-discovery (each agent profile suggests its canonical paths). Reads the identity sidecar each incarnation already emits (Kiro CLI's{uuid}.jsoncarriesagent_id.name+session_created_reason; IDE'ssession.jsoncarriesagentMode+workspacePaths) to stamp provenance.patterns/{locale}.yaml— a loader + fallback, applied to the agent dimension. A profile declares format detection, discovery globs, block→role, where the text lives, tool-result schema, injected markers. The irreducible nested-navigation stays in named code hooks (not everything fits in YAML). Adding an agent becomes writing a YAML + tests, not editing the parser.config/locale.py.Two entry doors this reconciles: use in any agent (harvesting local agents, C1–C3) and bring your memory (importing foreign systems like mem0/letta — the importer path). They share C2 (foreign format → our schema). This is also the discovery/parsing layer of the portability stack from #1364.
Why one design and not three PRs of guesswork: these were three separate RFCs (source-identity, kiro-sessions, design-extraction). Measuring showed they're the same problem. But the execution stays small and measured — first increment: extract one agent's (Kiro) hardcoded rules into
harvest/agents/kiro.yaml, parser code untouched, with the #1350 coverage instrument as a golden test proving byte-identical output. No behavior change, trivial compat, reversible. Later increments (path interpreter, multi-source registry, heuristics, identity) each as their own small PR.Full draft in the fork RFC. Does the layered shape work for you, and is the Kiro-to-YAML extraction the right first increment? Related: #1364 (portability stack), #1346 (design-extraction, now C3).
All reactions