Repository navigation
Embedding the Engine
SymForge is a product and a library. This page is for the second case: a Rust application that wants SymForge's indexing, search, editing and git core in-process, with no MCP transport and no subprocess. Everything here is true as of 11.7.0.
| You want | Use |
|---|---|
| An agent client (Claude Code, Codex, Cursor, ...) talking to SymForge | The MCP server. See Tool-Reference. |
| SymForge's engine inside your own Rust process, typed results, no wire |
symforge::embed (this page) |
| The whole SymForge CLI and server, launched from your own binary |
symforge::server_api::run (below) |
symforge = { version = "11", default-features = false, features = ["embed"] }default-features = false drops the server feature, which is the default. That
is not cosmetic: the daemon, sidecar, protocol server, CLI, watcher, analytics and
version registry are not compiled, and neither are their dependencies (rmcp,
axum, clap, reqwest, socket2, notify, tracing-subscriber, tokio/net).
Server modules live under #[cfg(feature = "server")], so server-side breakage
structurally cannot reach an embedder. The embed feature itself adds
libgit2-sys.
| Feature | Gives you |
|---|---|
embed |
symforge::embed and symforge::embed::parity
|
server (default) |
symforge::server_api and the MCP server |
__test-internals |
Not supported. A repo-internal test door; never enable it downstream. |
symforge::embed and symforge::server_api are the only public doors. The
toolchain is pinned to Rust 1.96 (rust-toolchain.toml), edition 2024.
CI builds the engine-only cell on every PR for glibc and for musl
(x86_64-unknown-linux-musl), runs clippy and the lib tests in that cell, and runs
the integration test tests/embed_bound_index.rs with --features embed.
The V11 cut was a breaking change to this facade. Depend on 11.x and open one
handle; the V10 raw surface (symforge::live_index, raw search functions,
snapshot loaders, update_file_from_disk, remove_file) is gone.
Acquire the process runtime, then open one embedded source per worktree root.
Opening admits the root and indexes it asynchronously; wait for the Current
phase before querying. Adapted from
public_embed_handle_indexes_refreshes_queries_and_joins in
tests/embed_bound_index.rs:
use std::time::{Duration, Instant};
use symforge::embed::{EmbeddedSourceSpec, ProcessIndexRuntime, SourceRuntimePhase};
let runtime = ProcessIndexRuntime::acquire()?;
let handle = runtime.open_embedded_source(
EmbeddedSourceSpec::current_worktree(repo_root.to_path_buf()),
)?;
let deadline = Instant::now() + Duration::from_secs(15);
while handle.runtime_view().phase != SourceRuntimePhase::Current {
assert!(Instant::now() < deadline, "source did not publish");
std::thread::sleep(Duration::from_millis(5));
}| Item | Signature / meaning |
|---|---|
ProcessIndexRuntime::acquire |
fn() -> Result<ProcessIndexRuntime, SourceRefusal> |
ProcessIndexRuntime::open_embedded_source |
fn(&self, EmbeddedSourceSpec) -> Result<EmbeddedSourceHandle, SourceRefusal> |
ProcessIndexRuntime::begin_shutdown |
Closes every source the runtime owns and joins its workers; returns a ShutdownReceipt whose wait(deadline) yields ShutdownReport { closed_sources, joined_workers }
|
EmbeddedSourceSpec::current_worktree |
fn(PathBuf) -> EmbeddedSourceSpec |
EmbeddedSourceHandle::runtime_view |
SourceRuntimeView { binding_identity, current_publication_identity, observer_epoch, phase, source_version } |
SourceRuntimePhase is Blocked, Current, Loading, Refreshing, Stopped or
Stopping. A second open of the same canonical source refuses until its handle
closes. Dropping the runtime stops the sources it opened.
Three typed requests, each answered from one pinned Current publication:
| Method | Request | Result |
|---|---|---|
search_symbols |
SymbolSearchRequest { query: Option<String>, path_prefix: Option<String>, limit: u32 } |
Claim<SymbolSearchResult>: matches: Vec<SymbolMatch { name, kind, path, start_line, end_line }>, truncated
|
search_text |
TextSearchRequest { query: String, path_prefix: Option<String>, limit: u32, case_sensitive: bool } |
Claim<TextSearchResult>: matches: Vec<TextMatch { path, line, byte_start, byte_end, preview }>, truncated
|
search_knowledge |
KnowledgeSearchRequest { query: String, path_prefix: Option<String>, limit: u32 } |
Claim<KnowledgeSearchResult>: matches: Vec<KnowledgeMatch>, truncated, withheld_count, withheld_reasons
|
Each returns Result<Claim<_>, SourceRefusal>. A KnowledgeMatch carries path,
heading_path, preview, content_hash, provenance_ids,
relationship_evidence, authority and coverage. Knowledge search covers
Markdown, plain text and config files.
use symforge::embed::{SymbolSearchRequest, TextSearchRequest};
let symbols = handle.search_symbols(&SymbolSearchRequest {
query: None,
path_prefix: None,
limit: u32::MAX,
})?;
assert!(symbols.value().matches.iter().any(|m| m.name == "Alpha"));
let text = handle.search_text(&TextSearchRequest {
query: "Alpha".to_string(),
path_prefix: Some("src".to_string()),
limit: u32::MAX,
case_sensitive: true,
})?;(From the same test. path_prefix: Some("src") matches src/ but not srcx/;
embed_path_prefix_src_excludes_srcx pins that.)
A Claim<T> is not a raw index dump. Read it through its accessors:
| Accessor | Returns |
|---|---|
value() |
&T |
provenance() |
&ClaimProvenance: identity(), kind_name(), authorities() (each an AtomicAuthority), authority_count()
|
operation() |
&OperationReceipt: identity(), canonical_argument_hash(), operation_kind(), schema_version()
|
evaluation() |
Option<&EvaluationProvenance> |
producing_runtime_identity() |
&str |
Two more read methods: index_census() returns Claim<IndexCensus> (total_files,
total_symbols, and files: Vec<CensusFile { path, language, symbol_count }>
sorted by path, including zero-symbol files, all from one publication), and the
infallible index_progress() returns IndexProgress { files_discovered, files_parsed, symbols_found } for the in-flight or last reload.
The source watches its own tree: a changed file moves the phase through
Refreshing back to Current with a higher source_version. Deletions are
observations, not commands; there is no remove_file.
To ask for a refresh explicitly, call request_refresh(). It returns
Result<RefreshTicket, SourceRefusal>; the ticket carries ticket_identity(),
requested_source_version() and an operation() receipt. Completion is
observable as a Current view whose source_version is greater than the one the
ticket names. request_refresh_with_reset() deletes the persisted snapshot scope
first, as MCP index_folder reset does.
Search never serves a stale or half-built index silently. While the source is not
Current, every typed search and index_census refuses with
SourceUnavailable, and the RetryAdvice says when to try again:
| Phase | RetryAdvice |
|---|---|
Loading, Refreshing
|
OnEvent (retry after the refresh publishes) |
Blocked |
Operator |
Stopping, Stopped, or the handle is closed |
Never |
Close with begin_close(), which always succeeds and returns a
SourceCloseReceipt. Its wait(deadline) returns SourceCloseReport { already_terminal, terminal_source_version } or a ReceiptWaitError. Waiting on
your own close from inside finalize refuses with WouldSelfWait instead of
deadlocking.
let info = symforge::embed::engine_info();
// info.version, info.snapshot_format_version,
// info.secret_policy_version, info.grammarsengine_info() returns the crate version, the snapshot format version (10 as of
11.7.0), the secret-policy version, and the lowercase name of every supported
grammar. All are compile-time constants: one call, no I/O. EngineInfo is
#[non_exhaustive].
Every fallible call returns a SourceRefusal. Match its kind; do not scrape the
Display string.
| Accessor | Returns |
|---|---|
kind() |
SourceRefusalKind: AdmissionUnavailable, InvalidSelection, SelectionUnavailable, SourceUnavailable
|
retry() |
RetryAdvice: Automatic (retry as is), Never (repetition cannot fix it), OnEvent (after a refresh or selection change), Operator (needs an operator action, such as a rebind) |
operation() |
The OperationReceipt; operation_kind() names one of the nine OperationKinds |
evidence_identity(), kind_name()
|
Stable identity strings |
Opening refuses InvalidSelection (retry Operator) when the root cannot be
bound, SelectionUnavailable (retry OnEvent) when the source is already open,
and AdmissionUnavailable or SourceUnavailable (retry Automatic) when
admission or the worker is unavailable.
symforge::embed::parity serves every MCP tool, resource and prompt natively,
rendered by the same shared engines the MCP handlers call. It is additive and
versioned separately from the frozen V11 core: parity::API_VERSION is 1. The
original V11 facade above stays source-compatible.
EmbeddedSourceHandle::query(&QueryRequest, QueryLimits) returns
Result<QueryClaim, QueryRefusal>. Each query captures one admitted publication;
lines are one-based and inclusive, byte spans zero-based and end-exclusive, and
returned source is byte-exact.
| MCP tool |
QueryRequest variant |
|---|---|
search_symbols, search_text, search_files, search_knowledge
|
SymbolSearch, TextSearch, FileSearch, SearchKnowledge
|
get_file_content |
FileContent, SourcePage
|
get_repo_map, get_file_context
|
RepoMap, FileContext
|
get_symbol, get_symbol_context, inspect_match
|
SymbolRead, SymbolContext, InspectMatch
|
find_references, find_dependents
|
ReferenceSearch, DependentSearch
|
validate_file_syntax |
Syntax; validate_syntax_from_disk works in any live phase |
analyze_file_impact |
FileImpact |
what_changed, diff_symbols, detect_impact
|
WhatChanged, DiffSymbols, DetectImpact
|
explore, conventions, context_inventory, investigation_suggest
|
Explore, Conventions, ContextInventory, InvestigationSuggest
|
edit_plan |
EditPlan { target } |
symforge_retrieve |
Retrieve { handle, offset } |
ask, symforge
|
Ask, Symforge
|
QueryLimits::default() is 100 rows and 65,536 bytes; requests above 10,000 rows
or 1 MiB, or with a zero bound, refuse. QueryRefusal::kind() returns a
QueryRefusalKind such as NotFound, AmbiguousSymbol, StalePublication,
Cancelled or DeadlineExceeded.
use symforge::embed::parity::search::TextSearchRequest;
use symforge::embed::parity::{QueryLimits, QueryRequest};
let claim = handle.query(
&QueryRequest::TextSearch(TextSearchRequest {
terms: Some(vec!["color".into(), "caller".into()]),
whole_word: Some(true),
glob: Some("**/*.rs".into()),
limit: Some(20),
..Default::default()
}),
QueryLimits::default(),
)?;(Adapted from
rich_text_search_applies_terms_scope_globs_words_and_limits_before_projection
in
tests/embed_search.rs.)
| Method | Purpose |
|---|---|
new_query_session(), new_query_session_with_limits(SessionCacheLimits)
|
A context history and retrieval cache bound to this source |
query_with_session(&request, limits, &session, Option<&OperationControl>) |
Session-aware query; a session from another source is refused |
query_with_control(&request, limits, &OperationControl) |
Host deadline and shared cancellation |
query_with_policy(&request, limits, QueryPolicy, session, control) |
Grants allow_derived_state_preparation
|
OperationControl::new(timeout) carries a deadline and a cancel flag;
cancel() stops work at the next safe checkpoint. A token-capped search keeps
the full result in the session cache, readable with QueryRequest::Retrieve
(search_token_caps_keep_full_shared_result_in_session_cache,
tests/embed_search.rs).
open_embedded_source_with_options(spec, EmbeddedOpenOptions) lets the trusted
host choose state placement (state), allow_protected_root,
reset_snapshot_state, replay_control_directory and
project_config_trust_mode (default LogOnly). These are Rust values, never
decoded from a wire request.
parity::host binds a source to a transport-neutral room. Wire requests carry no
root, room authority or state path; the host's broker picks the room.
- Build
HostRights(start fromHostRights::read_only();edit,refresh,checkpoint,curate_knowledge, secret rights and others are opt-in). -
HostRoomGrant::new(room_id, absolute_root, source_scope, rights). -
HostRoom::open(runtime, grant, HostRoomConfig { room_id, limits }), orHostRoom::open_in_runtime(&HostRuntimeOwner, ...)to share one runtime across rooms. -
room.dispatch(&HostRequest, &control)for typed calls, orroom.dispatch_wire(&json_bytes, &control)for serialized ones.
HostRequest |
MCP surface |
|---|---|
Query { request, limits } |
The query tools above |
StatusReport, HealthReport, HealthCompact
|
status, health, health_compact (Status, Health are the plain typed forms) |
RefreshWith(HostRefreshRequest { reset, idempotency_key }), Refresh
|
index_folder on the bound source |
Checkpoint { verify_after_write, export_artifact } |
checkpoint_now |
Edit { request, operation_key }, SymforgeEdit
|
The edit tools, symforge_edit
|
Knowledge { request, operation_key } |
review_knowledge, curate_knowledge
|
InspectSecrets, PreviewSecretRemediation, ApplySecretRemediation
|
secret_remediate |
Resource(HostResourceRequest) |
The 6 resources and 4 resource templates |
Prompt(HostPromptRequest) |
The 8 prompts (Admin needs a host-supplied HostPromptContext) |
Catalog |
Operations, resources, templates, prompts and rights for this room |
HostLimits::default() caps requests at 64 KiB, responses at 1 MiB, queries at
1,000 rows and 256 KiB, and duration at 120 s, and bounds what a room will index
at 100,000 files and 1 GiB. Refusals are HostRefusal with a
HostRefusalKind. The room-opening and wire-dispatch pattern is in
tests/embed_host.rs.
Edits are plan, preview, apply. A plan binds the exact publication, file image and
symbol; apply needs a separate write authority the host mints after its own rights
check. EditError does not implement std::error::Error, so match it rather
than using ?. Adapted from
guarded_replace_previews_applies_replays_and_refuses_stale_generation in
tests/embed_mutation.rs:
use std::sync::{Arc, atomic::AtomicBool};
use symforge::embed::parity::edit::{EditApplyAuthority, EditTarget, ReplaceRequest};
let plan = handle.edit_plan(&EditTarget {
path: "src/lib.rs".to_string(),
name: "target".to_string(),
kind: None,
symbol_line: None,
}).expect("plan exact target");
let request = ReplaceRequest {
guard: plan.guard.clone(),
new_body: "pub fn target() -> u32 { 2 }\n".to_string(),
};
let preview = handle.preview_replace(&request).expect("preview"); // writes nothing
let authority = EditApplyAuthority::for_source_root(
std::fs::canonicalize(repo_root).expect("canonical root"),
"my-room/source".to_string(),
Arc::new(AtomicBool::new(false)), // cancel flag
)
.expect("host authority");
let applied = handle
.apply_replace(&request, &authority, "operation-1")
.expect("guarded apply");The same shape covers preview_insert/apply_insert, preview_delete/
apply_delete, preview_edit_within/apply_edit_within, and the batch lanes
(preview_batch_edit, apply_batch_edit, preview_batch_insert,
apply_batch_insert, plan_batch_rename, preview_batch_rename,
apply_batch_rename). Repeating an apply with the same operation key replays
(applied.replayed is true) without writing; applying a plan whose publication
has moved refuses with EditErrorKind::StaleGeneration.
MCP's working_directory maps to handle.route_edit(path, working_directory, &admitted), where admitted is a slice of AdmittedEditTarget { handle, authority }. That list is the whole authority boundary: a directory that is not
the bound root or an admitted worktree of the same repository refuses before
anything is written, and routing never opens another root. The returned
EditRoute reports working_directory, rerouted, target_path and
indexed_path, and renders MCP's reroute suffix
(tests/embed_edit_route.rs).
parity::replay::ReplayStore is the replay lane for a trusted host's own
effects: reserve a ReplayKey with a RequestFingerprint, mark_started
before the effect, then complete, fail, mark_uncertain or reconcile. A crash leaves Started,
which never becomes executable again by age or retry. The store keeps only an
outcome kind, a response digest and a post-image digest, never a response body.
Contract:
docs/contracts/embed-replay-v1.md.
The census of record is
docs/reviews/2026-10-10-embed-parity-gap-matrix.md.
Its final row census marks every tool, resource and prompt FULL and lists
these exceptions:
- Edit replay text. A same-key embedded replay answers with a replay notice naming the tool, path and verified post-image digest, not the original answer text, because the replay store keeps digests only. Outcome class and written bytes match MCP.
- Tee path of a routed edit. Tee paths are timestamped on both sides. A rerouted embedded edit tees into the routed worktree's own admitted state; MCP tees into the indexed project's state. Both answers report the preserved original the same way.
-
Process environment. Embed reads no process-global policy:
SYMFORGE_WORKTREE_AWAREand the CI trust overrideSYMFORGE_TRUST_PROJECT_CONFIGare not read; the host's admitted list andEmbeddedOpenOptionsare the policy. The trust mode is the typed optionEmbeddedOpenOptions::project_config_trust_mode(defaultLogOnly;Enforcerefuses every edit lane with MCP's refusal text). -
Routed
batch_renameimpact footer. A routed rename reads its impact footer from the routed worktree's post-image index; MCP reads the bound index after reindexing the written files. They agree whenever the untouched files of the two worktrees agree.
Two further limits come from libgit2 itself and are recorded in the description of PR #780, with one test kept ignored for them:
- Global git config include. A malformed include in the user's global git config still refuses git preparation: libgit2 loads global config in every public open.
- System gitattributes. System gitattributes still apply: libgit2 reads system attributes unconditionally.
Snapshot restore is engine-internal; you do not call a loader. A snapshot on disk
may speed up re-proof but never confers authority by itself. The snapshot format is
10 as of 11.7.0, and a snapshot from an older format is an untrusted seed: it
never restores. A copy is kept under .symforge/v11/quarantine/index-snapshots/,
and the first snapshot write moves the original out of index.bin into
.symforge/quarantine/index-snapshots/, so a downgrade must restore one of those
copies. See Runtime-Model for the lifecycle.
With the default server feature, symforge::server_api::run(args: Vec<OsString>) -> Result<ServerExit, ServerBootstrapError> runs the whole
SymForge CLI (argv[0] included). The symforge binary is a shim over it.
-
ServerExitisSuccessorRefusedToStart. -
ServerBootstrapErroris opaque: no public fields, so it cannot be built or matched exhaustively.Displaygives the full cause chain.
symforge::embed is semver-public. A breaking change to any name or signature in
it is a MAJOR bump; additive changes are MINOR. This is enforced at compile time:
the contract test module in
src/embed.rs
names every contracted type and binds acquire, open_embedded_source,
current_worktree, engine_info, index_census, index_progress and
search_knowledge to their full signatures as fn pointers. A rename, removal
or signature change fails SymForge's own embed test suite before a downstream
integrator sees it. server_api carries its own shape test in
src/server_api.rs.
symforge::embed::parity is versioned by API_VERSION, not by that contract
module.
-
CHANGELOG.mdopens with a hand-maintained Embedder API (embed) changes section, above the release entries. It tracks only changes to this facade. -
docs/migrations/v11-index-lifecycle.mdis the V10 to V11 guide: removed exports, the replacement atoms, snapshot and rollback constraints, and a compile-fix crib sheet.
Need the short version? The README links each topic to its page here.
Install and client setup: Install-and-Client-Setup