-
Notifications
You must be signed in to change notification settings - Fork 2
relayburn-sdk: port the ingest verb #302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,106 @@ | ||
| //! Ingest verb — async wrapper over `relayburn_ingest::ingest_all`. | ||
| //! Filled in by the follow-up to #246 PR1. | ||
| //! Ingest verb — async wrapper over [`relayburn_ingest::ingest_all`]. | ||
| //! | ||
| //! Both an `impl LedgerHandle` method (`async fn ingest`) and a free | ||
| //! function form (`pub async fn ingest`) live here. | ||
| //! Mirrors the TS `ingest` verb in `packages/sdk/index.js`. The Rust port | ||
| //! threads the ledger location through [`crate::Ledger::open`] explicitly | ||
| //! instead of swapping `RELAYBURN_HOME`, so embeddings can run against | ||
| //! multiple ledgers in the same process. | ||
|
|
||
| // TODO(#246): port the ingest verb from `packages/sdk/index.js`. | ||
| use std::path::PathBuf; | ||
|
|
||
| use relayburn_ingest::{ingest_all, IngestOptions as RawIngestOptions, IngestReport, IngestRoots}; | ||
|
|
||
| use crate::{Ledger, LedgerHandle, LedgerOpenOptions}; | ||
|
|
||
| /// Sink for short orchestration progress strings (one per phase). | ||
| pub type ProgressSink = Box<dyn Fn(&str) + Send + Sync>; | ||
|
|
||
| /// Sink for content-capture gap warnings emitted during ingest. | ||
| pub type WarnSink = Box<dyn Fn(&str) + Send + Sync>; | ||
|
|
||
| /// SDK-level options for the [`ingest`] verb. Mirrors the TS shape but | ||
| /// uses Rust-friendly types (`PathBuf`, boxed sinks). | ||
| /// | ||
| /// `ledger_home` is only consulted by the free [`ingest`] function (which | ||
| /// opens its own [`LedgerHandle`]); the [`LedgerHandle::ingest`] method | ||
| /// uses the already-open ledger and ignores it. | ||
| #[derive(Default)] | ||
| pub struct IngestOptions { | ||
| /// Override for `$RELAYBURN_HOME`. Forwarded to [`LedgerOpenOptions::home`] | ||
| /// when the free function opens a ledger. | ||
| pub ledger_home: Option<PathBuf>, | ||
| /// Per-harness session-store roots. Defaults to scanning the developer's | ||
| /// home dir (`~/.claude/projects`, `~/.codex/sessions`, | ||
| /// `~/.local/share/opencode/storage`); tests must inject explicit roots. | ||
| pub roots: IngestRoots, | ||
| /// Optional sink for short orchestration progress strings. | ||
| pub on_progress: Option<ProgressSink>, | ||
| /// Optional sink for content-capture gap warnings. | ||
| pub on_warn: Option<WarnSink>, | ||
| } | ||
|
|
||
| impl IngestOptions { | ||
| /// Convert to the lower-crate options struct, consuming the closure | ||
| /// sinks. `ledger_home` is dropped here — it is consumed at ledger-open | ||
| /// time and is not part of the lower-crate API. | ||
| fn into_raw(self) -> RawIngestOptions { | ||
| RawIngestOptions { | ||
| on_progress: self.on_progress, | ||
| on_warn: self.on_warn, | ||
| roots: self.roots, | ||
| } | ||
| } | ||
| } | ||
|
|
||
| impl LedgerHandle { | ||
| /// Run [`ingest_all`] against this ledger handle. Returns the merged | ||
| /// per-harness report. | ||
| pub async fn ingest(&mut self, opts: IngestOptions) -> anyhow::Result<IngestReport> { | ||
| let raw = opts.into_raw(); | ||
| ingest_all(&mut self.inner, &raw).await | ||
| } | ||
| } | ||
|
|
||
| /// Free-function form of the ingest verb. Opens a fresh ledger using | ||
| /// `opts.ledger_home`, runs [`ingest_all`], and returns the report. | ||
| pub async fn ingest(opts: IngestOptions) -> anyhow::Result<IngestReport> { | ||
| let mut handle = Ledger::open(LedgerOpenOptions { | ||
| home: opts.ledger_home.clone(), | ||
| ..Default::default() | ||
| })?; | ||
| handle.ingest(opts).await | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::*; | ||
| use tempfile::TempDir; | ||
|
|
||
| #[tokio::test] | ||
| async fn ingest_with_empty_roots_returns_zero_report() { | ||
| let home = TempDir::new().expect("home tmp"); | ||
| let claude = TempDir::new().expect("claude tmp"); | ||
| let codex = TempDir::new().expect("codex tmp"); | ||
| let opencode = TempDir::new().expect("opencode tmp"); | ||
|
|
||
| // Point `RELAYBURN_HOME` at the temp dir so `cleanup_stale_pending_stamps` | ||
| // and `load_config` (called inside ingest_all) don't touch the real | ||
| // `~/.relayburn`. Set before any ledger-open call. | ||
| std::env::set_var("RELAYBURN_HOME", home.path()); | ||
|
|
||
| let opts = IngestOptions { | ||
| ledger_home: Some(home.path().to_path_buf()), | ||
| roots: IngestRoots { | ||
| claude_projects_dir: Some(claude.path().to_path_buf()), | ||
| codex_sessions_dir: Some(codex.path().to_path_buf()), | ||
| opencode_storage_dir: Some(opencode.path().to_path_buf()), | ||
| }, | ||
| on_progress: None, | ||
| on_warn: None, | ||
| }; | ||
|
|
||
| let report = ingest(opts).await.expect("ingest"); | ||
| assert_eq!(report.scanned_sessions, 0); | ||
| assert_eq!(report.ingested_sessions, 0); | ||
| assert_eq!(report.appended_turns, 0); | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
into_rawdropsledger_home, butingest_allstill performs home-scoped side effects viacleanup_stale_pending_stamps()andload_config()(incrates/relayburn-ingest/src/ingest.rs), both of which resolve paths fromRELAYBURN_HOME. As a result, callingingestwith a non-defaultledger_homecan clean pending stamps and read content settings from the wrong ledger, violating the SDK contract for multi-ledger embeddings and potentially mutating another ledger’s state.Useful? React with 👍 / 👎.