Skip to content

MCP-native harvest: post a transcript instead of rsyncing session files - #116

Merged
raphasouthall merged 1 commit into
mainfrom
fix/115-mcp-native-harvest
Aug 25, 2026
Merged

MCP-native harvest: post a transcript instead of rsyncing session files#116
raphasouthall merged 1 commit into
mainfrom
fix/115-mcp-native-harvest

Conversation

@raphasouthall

Copy link
Copy Markdown
Owner

Part of #115.

What

vault_harvest_transcript(transcript, session_id, source_agent, dry_run) — the client posts its own session text and the server harvests it, so capture no longer depends on the server being able to read the client's filesystem. Adds OmpProvider for ~/.omp/agent/sessions/*/*.jsonl so the file-scan path and backfills understand omp sessions too.

How

The classify/redact/dedup/save loop moved verbatim out of harvest_sessions into _harvest_messages, which both entry points call. harvest_sessions keeps session discovery, the mtime state filter and the report; its public behaviour and report shape are unchanged and no existing test was touched. Because redact_secrets sits inside the shared seam, ahead of the dedup check, the #113 contract applies to the new unattended write path by construction rather than by a second call site.

harvest_transcript writes the posted text to a temp file and hands it to the named provider's existing extract_messages, so no provider grew a second entry point. The temp file is unlinked in a finally covering both the write and the parse.

No server-side chunk assembly. A transcript over MAX_TRANSCRIPT_BYTES (4 MiB) returns an error telling the client to split on newline boundaries: JSONL chunks are whole messages, each chunk harvests independently, and the cosine dedup absorbs overlap. The re-post guard reuses harvest_state under an mcp:<source_agent>:<session_id> key holding the transcript's sha256, so a retried POST is a no-op; the mtime comparison in harvest_sessions is now numeric-only so the two key namespaces cannot be confused.

OmpProvider keeps only user and assistant roles — toolResult also carries text parts, and including them drowns the pre-filter in tool output — and only text content parts, dropping thinking and toolCall.

Gate

uv run ruff check src/ tests/ exit 0. uv run pytest -q exit 0, 804 passed (792 existing + 12 new), reproduced cold after the build.

Live verify

Against real on-disk omp sessions, read-only: find_sessions(3) returned 3, and the newest parsed to 86 messages (31 user, 55 assistant) with all 166 toolResult messages and every thinking/toolCall part excluded; a dry-run harvest of that file produced 8 insights. Through the tool registry, a dry-run post of a Claude-shaped line returned counts {'bug': 1}, and an unknown source_agent returned an error naming the six registered providers. End-to-end verification against the deployed server from both a Claude Code and an omp client follows the merge, and the interim rsync timer is retired only after that.

Known gap

The omp tree also holds depth-3 <project>/<session>/<AgentName>.jsonl subagent transcripts. The glob is deliberately depth 2, so those are not picked up by the file-scan path; they are largely tool chatter and would multiply the scan. A client can still post one explicitly.

Harvest could only see session files on the machine running the server,
so capture depended on rsyncing transcripts to it first.

Extract the classify/redact/dedup/save loop out of harvest_sessions into
_harvest_messages, and add harvest_transcript on top of it: the client
posts its own session text, the named provider parses it, and the rest of
the path is unchanged. Redaction sits inside the shared seam, so the
issue #113 contract holds for both entry points by construction.

Chunking stays client-side. JSONL is line-oriented, so a chunk split on a
newline yields whole messages, each chunk is harvested independently, and
the 0.88 cosine dedup absorbs any overlap. A re-post guard keyed
mcp:<source_agent>:<session_id> makes an identical retry a no-op.

Also add OmpProvider for ~/.omp/agent/sessions/*/*.jsonl so the file-scan
path and backfills understand omp sessions. Its toolResult messages carry
raw tool output and are excluded, as are thinking and toolCall parts.

Part of #115
@raphasouthall
raphasouthall merged commit 22f82ab into main Aug 25, 2026
5 checks passed
@raphasouthall
raphasouthall deleted the fix/115-mcp-native-harvest branch August 25, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant