MCP-native harvest: post a transcript instead of rsyncing session files - #116
Merged
Conversation
Harvest could only see session files on the machine running the server, so capture depended on rsyncing transcripts to it first. Extract the classify/redact/dedup/save loop out of harvest_sessions into _harvest_messages, and add harvest_transcript on top of it: the client posts its own session text, the named provider parses it, and the rest of the path is unchanged. Redaction sits inside the shared seam, so the issue #113 contract holds for both entry points by construction. Chunking stays client-side. JSONL is line-oriented, so a chunk split on a newline yields whole messages, each chunk is harvested independently, and the 0.88 cosine dedup absorbs any overlap. A re-post guard keyed mcp:<source_agent>:<session_id> makes an identical retry a no-op. Also add OmpProvider for ~/.omp/agent/sessions/*/*.jsonl so the file-scan path and backfills understand omp sessions. Its toolResult messages carry raw tool output and are excluded, as are thinking and toolCall parts. Part of #115
This was referenced Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #115.
What
vault_harvest_transcript(transcript, session_id, source_agent, dry_run)— the client posts its own session text and the server harvests it, so capture no longer depends on the server being able to read the client's filesystem. AddsOmpProviderfor~/.omp/agent/sessions/*/*.jsonlso the file-scan path and backfills understand omp sessions too.How
The classify/redact/dedup/save loop moved verbatim out of
harvest_sessionsinto_harvest_messages, which both entry points call.harvest_sessionskeeps session discovery, the mtime state filter and the report; its public behaviour and report shape are unchanged and no existing test was touched. Becauseredact_secretssits inside the shared seam, ahead of the dedup check, the #113 contract applies to the new unattended write path by construction rather than by a second call site.harvest_transcriptwrites the posted text to a temp file and hands it to the named provider's existingextract_messages, so no provider grew a second entry point. The temp file is unlinked in afinallycovering both the write and the parse.No server-side chunk assembly. A transcript over
MAX_TRANSCRIPT_BYTES(4 MiB) returns an error telling the client to split on newline boundaries: JSONL chunks are whole messages, each chunk harvests independently, and the cosine dedup absorbs overlap. The re-post guard reusesharvest_stateunder anmcp:<source_agent>:<session_id>key holding the transcript's sha256, so a retried POST is a no-op; the mtime comparison inharvest_sessionsis now numeric-only so the two key namespaces cannot be confused.OmpProviderkeeps onlyuserandassistantroles —toolResultalso carries text parts, and including them drowns the pre-filter in tool output — and onlytextcontent parts, droppingthinkingandtoolCall.Gate
uv run ruff check src/ tests/exit 0.uv run pytest -qexit 0, 804 passed (792 existing + 12 new), reproduced cold after the build.Live verify
Against real on-disk omp sessions, read-only:
find_sessions(3)returned 3, and the newest parsed to 86 messages (31 user, 55 assistant) with all 166toolResultmessages and everythinking/toolCallpart excluded; a dry-run harvest of that file produced 8 insights. Through the tool registry, a dry-run post of a Claude-shaped line returnedcounts {'bug': 1}, and an unknownsource_agentreturned an error naming the six registered providers. End-to-end verification against the deployed server from both a Claude Code and an omp client follows the merge, and the interim rsync timer is retired only after that.Known gap
The omp tree also holds depth-3
<project>/<session>/<AgentName>.jsonlsubagent transcripts. The glob is deliberately depth 2, so those are not picked up by the file-scan path; they are largely tool chatter and would multiply the scan. A client can still post one explicitly.