Skip to content

Embedding the Engine

special-place-administrator edited this page Oct 11, 2026 · 2 revisions

Embedding the Engine

SymForge is a product and a library. This page is for the second case: a Rust application that wants SymForge's indexing, search, editing and git core in-process, with no MCP transport and no subprocess. Everything here is true as of 11.7.0.

When to embed

You want Use
An agent client (Claude Code, Codex, Cursor, ...) talking to SymForge The MCP server. See Tool-Reference.
SymForge's engine inside your own Rust process, typed results, no wire symforge::embed (this page)
The whole SymForge CLI and server, launched from your own binary symforge::server_api::run (below)

Build and dependency

symforge = { version = "11", default-features = false, features = ["embed"] }

default-features = false drops the server feature, which is the default. That is not cosmetic: the daemon, sidecar, protocol server, CLI, watcher, analytics and version registry are not compiled, and neither are their dependencies (rmcp, axum, clap, reqwest, socket2, notify, tracing-subscriber, tokio/net). Server modules live under #[cfg(feature = "server")], so server-side breakage structurally cannot reach an embedder. The embed feature itself adds libgit2-sys.

Feature Gives you
embed symforge::embed and symforge::embed::parity
server (default) symforge::server_api and the MCP server
__test-internals Not supported. A repo-internal test door; never enable it downstream.

symforge::embed and symforge::server_api are the only public doors. The toolchain is pinned to Rust 1.96 (rust-toolchain.toml), edition 2024.

CI builds the engine-only cell on every PR for glibc and for musl (x86_64-unknown-linux-musl), runs clippy and the lib tests in that cell, and runs the integration test tests/embed_bound_index.rs with --features embed.

The V11 cut was a breaking change to this facade. Depend on 11.x and open one handle; the V10 raw surface (symforge::live_index, raw search functions, snapshot loaders, update_file_from_disk, remove_file) is gone.

Open a source

Acquire the process runtime, then open one embedded source per worktree root. Opening admits the root and indexes it asynchronously; wait for the Current phase before querying. Adapted from public_embed_handle_indexes_refreshes_queries_and_joins in tests/embed_bound_index.rs:

use std::time::{Duration, Instant};
use symforge::embed::{EmbeddedSourceSpec, ProcessIndexRuntime, SourceRuntimePhase};

let runtime = ProcessIndexRuntime::acquire()?;
let handle = runtime.open_embedded_source(
    EmbeddedSourceSpec::current_worktree(repo_root.to_path_buf()),
)?;

let deadline = Instant::now() + Duration::from_secs(15);
while handle.runtime_view().phase != SourceRuntimePhase::Current {
    assert!(Instant::now() < deadline, "source did not publish");
    std::thread::sleep(Duration::from_millis(5));
}
Item Signature / meaning
ProcessIndexRuntime::acquire fn() -> Result<ProcessIndexRuntime, SourceRefusal>
ProcessIndexRuntime::open_embedded_source fn(&self, EmbeddedSourceSpec) -> Result<EmbeddedSourceHandle, SourceRefusal>
ProcessIndexRuntime::begin_shutdown Closes every source the runtime owns and joins its workers; returns a ShutdownReceipt whose wait(deadline) yields ShutdownReport { closed_sources, joined_workers }
EmbeddedSourceSpec::current_worktree fn(PathBuf) -> EmbeddedSourceSpec
EmbeddedSourceHandle::runtime_view SourceRuntimeView { binding_identity, current_publication_identity, observer_epoch, phase, source_version }

SourceRuntimePhase is Blocked, Current, Loading, Refreshing, Stopped or Stopping. A second open of the same canonical source refuses until its handle closes. Dropping the runtime stops the sources it opened.

Typed search and claims

Three typed requests, each answered from one pinned Current publication:

Method Request Result
search_symbols SymbolSearchRequest { query: Option<String>, path_prefix: Option<String>, limit: u32 } Claim<SymbolSearchResult>: matches: Vec<SymbolMatch { name, kind, path, start_line, end_line }>, truncated
search_text TextSearchRequest { query: String, path_prefix: Option<String>, limit: u32, case_sensitive: bool } Claim<TextSearchResult>: matches: Vec<TextMatch { path, line, byte_start, byte_end, preview }>, truncated
search_knowledge KnowledgeSearchRequest { query: String, path_prefix: Option<String>, limit: u32 } Claim<KnowledgeSearchResult>: matches: Vec<KnowledgeMatch>, truncated, withheld_count, withheld_reasons

Each returns Result<Claim<_>, SourceRefusal>. A KnowledgeMatch carries path, heading_path, preview, content_hash, provenance_ids, relationship_evidence, authority and coverage. Knowledge search covers Markdown, plain text and config files.

use symforge::embed::{SymbolSearchRequest, TextSearchRequest};

let symbols = handle.search_symbols(&SymbolSearchRequest {
    query: None,
    path_prefix: None,
    limit: u32::MAX,
})?;
assert!(symbols.value().matches.iter().any(|m| m.name == "Alpha"));

let text = handle.search_text(&TextSearchRequest {
    query: "Alpha".to_string(),
    path_prefix: Some("src".to_string()),
    limit: u32::MAX,
    case_sensitive: true,
})?;

(From the same test. path_prefix: Some("src") matches src/ but not srcx/; embed_path_prefix_src_excludes_srcx pins that.)

A Claim<T> is not a raw index dump. Read it through its accessors:

Accessor Returns
value() &T
provenance() &ClaimProvenance: identity(), kind_name(), authorities() (each an AtomicAuthority), authority_count()
operation() &OperationReceipt: identity(), canonical_argument_hash(), operation_kind(), schema_version()
evaluation() Option<&EvaluationProvenance>
producing_runtime_identity() &str

Two more read methods: index_census() returns Claim<IndexCensus> (total_files, total_symbols, and files: Vec<CensusFile { path, language, symbol_count }> sorted by path, including zero-symbol files, all from one publication), and the infallible index_progress() returns IndexProgress { files_discovered, files_parsed, symbols_found } for the in-flight or last reload.

Refresh, progress and close

The source watches its own tree: a changed file moves the phase through Refreshing back to Current with a higher source_version. Deletions are observations, not commands; there is no remove_file.

To ask for a refresh explicitly, call request_refresh(). It returns Result<RefreshTicket, SourceRefusal>; the ticket carries ticket_identity(), requested_source_version() and an operation() receipt. Completion is observable as a Current view whose source_version is greater than the one the ticket names. request_refresh_with_reset() deletes the persisted snapshot scope first, as MCP index_folder reset does.

Search never serves a stale or half-built index silently. While the source is not Current, every typed search and index_census refuses with SourceUnavailable, and the RetryAdvice says when to try again:

Phase RetryAdvice
Loading, Refreshing OnEvent (retry after the refresh publishes)
Blocked Operator
Stopping, Stopped, or the handle is closed Never

Close with begin_close(), which always succeeds and returns a SourceCloseReceipt. Its wait(deadline) returns SourceCloseReport { already_terminal, terminal_source_version } or a ReceiptWaitError. Waiting on your own close from inside finalize refuses with WouldSelfWait instead of deadlocking.

Engine identity

let info = symforge::embed::engine_info();
// info.version, info.snapshot_format_version,
// info.secret_policy_version, info.grammars

engine_info() returns the crate version, the snapshot format version (10 as of 11.7.0), the secret-policy version, and the lowercase name of every supported grammar. All are compile-time constants: one call, no I/O. EngineInfo is #[non_exhaustive].

Typed refusals

Every fallible call returns a SourceRefusal. Match its kind; do not scrape the Display string.

Accessor Returns
kind() SourceRefusalKind: AdmissionUnavailable, InvalidSelection, SelectionUnavailable, SourceUnavailable
retry() RetryAdvice: Automatic (retry as is), Never (repetition cannot fix it), OnEvent (after a refresh or selection change), Operator (needs an operator action, such as a rebind)
operation() The OperationReceipt; operation_kind() names one of the nine OperationKinds
evidence_identity(), kind_name() Stable identity strings

Opening refuses InvalidSelection (retry Operator) when the root cannot be bound, SelectionUnavailable (retry OnEvent) when the source is already open, and AdmissionUnavailable or SourceUnavailable (retry Automatic) when admission or the worker is unavailable.

MCP parity: embed::parity

symforge::embed::parity serves every MCP tool, resource and prompt natively, rendered by the same shared engines the MCP handlers call. It is additive and versioned separately from the frozen V11 core: parity::API_VERSION is 1. The original V11 facade above stays source-compatible.

Queries

EmbeddedSourceHandle::query(&QueryRequest, QueryLimits) returns Result<QueryClaim, QueryRefusal>. Each query captures one admitted publication; lines are one-based and inclusive, byte spans zero-based and end-exclusive, and returned source is byte-exact.

MCP tool QueryRequest variant
search_symbols, search_text, search_files, search_knowledge SymbolSearch, TextSearch, FileSearch, SearchKnowledge
get_file_content FileContent, SourcePage
get_repo_map, get_file_context RepoMap, FileContext
get_symbol, get_symbol_context, inspect_match SymbolRead, SymbolContext, InspectMatch
find_references, find_dependents ReferenceSearch, DependentSearch
validate_file_syntax Syntax; validate_syntax_from_disk works in any live phase
analyze_file_impact FileImpact
what_changed, diff_symbols, detect_impact WhatChanged, DiffSymbols, DetectImpact
explore, conventions, context_inventory, investigation_suggest Explore, Conventions, ContextInventory, InvestigationSuggest
edit_plan EditPlan { target }
symforge_retrieve Retrieve { handle, offset }
ask, symforge Ask, Symforge

QueryLimits::default() is 100 rows and 65,536 bytes; requests above 10,000 rows or 1 MiB, or with a zero bound, refuse. QueryRefusal::kind() returns a QueryRefusalKind such as NotFound, AmbiguousSymbol, StalePublication, Cancelled or DeadlineExceeded.

use symforge::embed::parity::search::TextSearchRequest;
use symforge::embed::parity::{QueryLimits, QueryRequest};

let claim = handle.query(
    &QueryRequest::TextSearch(TextSearchRequest {
        terms: Some(vec!["color".into(), "caller".into()]),
        whole_word: Some(true),
        glob: Some("**/*.rs".into()),
        limit: Some(20),
        ..Default::default()
    }),
    QueryLimits::default(),
)?;

(Adapted from rich_text_search_applies_terms_scope_globs_words_and_limits_before_projection in tests/embed_search.rs.)

Sessions and cancellation

Method Purpose
new_query_session(), new_query_session_with_limits(SessionCacheLimits) A context history and retrieval cache bound to this source
query_with_session(&request, limits, &session, Option<&OperationControl>) Session-aware query; a session from another source is refused
query_with_control(&request, limits, &OperationControl) Host deadline and shared cancellation
query_with_policy(&request, limits, QueryPolicy, session, control) Grants allow_derived_state_preparation

OperationControl::new(timeout) carries a deadline and a cancel flag; cancel() stops work at the next safe checkpoint. A token-capped search keeps the full result in the session cache, readable with QueryRequest::Retrieve (search_token_caps_keep_full_shared_result_in_session_cache, tests/embed_search.rs).

open_embedded_source_with_options(spec, EmbeddedOpenOptions) lets the trusted host choose state placement (state), allow_protected_root, reset_snapshot_state, replay_control_directory and project_config_trust_mode (default LogOnly). These are Rust values, never decoded from a wire request.

Host rooms: tools, resources and prompts

parity::host binds a source to a transport-neutral room. Wire requests carry no root, room authority or state path; the host's broker picks the room.

  1. Build HostRights (start from HostRights::read_only(); edit, refresh, checkpoint, curate_knowledge, secret rights and others are opt-in).
  2. HostRoomGrant::new(room_id, absolute_root, source_scope, rights).
  3. HostRoom::open(runtime, grant, HostRoomConfig { room_id, limits }), or HostRoom::open_in_runtime(&HostRuntimeOwner, ...) to share one runtime across rooms.
  4. room.dispatch(&HostRequest, &control) for typed calls, or room.dispatch_wire(&json_bytes, &control) for serialized ones.
HostRequest MCP surface
Query { request, limits } The query tools above
StatusReport, HealthReport, HealthCompact status, health, health_compact (Status, Health are the plain typed forms)
RefreshWith(HostRefreshRequest { reset, idempotency_key }), Refresh index_folder on the bound source
Checkpoint { verify_after_write, export_artifact } checkpoint_now
Edit { request, operation_key }, SymforgeEdit The edit tools, symforge_edit
Knowledge { request, operation_key } review_knowledge, curate_knowledge
InspectSecrets, PreviewSecretRemediation, ApplySecretRemediation secret_remediate
Resource(HostResourceRequest) The 6 resources and 4 resource templates
Prompt(HostPromptRequest) The 8 prompts (Admin needs a host-supplied HostPromptContext)
Catalog Operations, resources, templates, prompts and rights for this room

HostLimits::default() caps requests at 64 KiB, responses at 1 MiB, queries at 1,000 rows and 256 KiB, and duration at 120 s, and bounds what a room will index at 100,000 files and 1 GiB. Refusals are HostRefusal with a HostRefusalKind. The room-opening and wire-dispatch pattern is in tests/embed_host.rs.

Editing

Edits are plan, preview, apply. A plan binds the exact publication, file image and symbol; apply needs a separate write authority the host mints after its own rights check. EditError does not implement std::error::Error, so match it rather than using ?. Adapted from guarded_replace_previews_applies_replays_and_refuses_stale_generation in tests/embed_mutation.rs:

use std::sync::{Arc, atomic::AtomicBool};
use symforge::embed::parity::edit::{EditApplyAuthority, EditTarget, ReplaceRequest};

let plan = handle.edit_plan(&EditTarget {
    path: "src/lib.rs".to_string(),
    name: "target".to_string(),
    kind: None,
    symbol_line: None,
}).expect("plan exact target");
let request = ReplaceRequest {
    guard: plan.guard.clone(),
    new_body: "pub fn target() -> u32 { 2 }\n".to_string(),
};
let preview = handle.preview_replace(&request).expect("preview"); // writes nothing

let authority = EditApplyAuthority::for_source_root(
    std::fs::canonicalize(repo_root).expect("canonical root"),
    "my-room/source".to_string(),
    Arc::new(AtomicBool::new(false)), // cancel flag
)
.expect("host authority");
let applied = handle
    .apply_replace(&request, &authority, "operation-1")
    .expect("guarded apply");

The same shape covers preview_insert/apply_insert, preview_delete/ apply_delete, preview_edit_within/apply_edit_within, and the batch lanes (preview_batch_edit, apply_batch_edit, preview_batch_insert, apply_batch_insert, plan_batch_rename, preview_batch_rename, apply_batch_rename). Repeating an apply with the same operation key replays (applied.replayed is true) without writing; applying a plan whose publication has moved refuses with EditErrorKind::StaleGeneration.

Admitted edit targets

MCP's working_directory maps to handle.route_edit(path, working_directory, &admitted), where admitted is a slice of AdmittedEditTarget { handle, authority }. That list is the whole authority boundary: a directory that is not the bound root or an admitted worktree of the same repository refuses before anything is written, and routing never opens another root. The returned EditRoute reports working_directory, rerouted, target_path and indexed_path, and renders MCP's reroute suffix (tests/embed_edit_route.rs).

Durable replay

parity::replay::ReplayStore is the replay lane for a trusted host's own effects: reserve a ReplayKey with a RequestFingerprint, mark_started before the effect, then complete, fail, mark_uncertain or reconcile. A crash leaves Started, which never becomes executable again by age or retry. The store keeps only an outcome kind, a response digest and a post-image digest, never a response body. Contract: docs/contracts/embed-replay-v1.md.

Known differences from MCP

The census of record is docs/reviews/2026-10-10-embed-parity-gap-matrix.md. Its final row census marks every tool, resource and prompt FULL and lists these exceptions:

  • Edit replay text. A same-key embedded replay answers with a replay notice naming the tool, path and verified post-image digest, not the original answer text, because the replay store keeps digests only. Outcome class and written bytes match MCP.
  • Tee path of a routed edit. Tee paths are timestamped on both sides. A rerouted embedded edit tees into the routed worktree's own admitted state; MCP tees into the indexed project's state. Both answers report the preserved original the same way.
  • Process environment. Embed reads no process-global policy: SYMFORGE_WORKTREE_AWARE and the CI trust override SYMFORGE_TRUST_PROJECT_CONFIG are not read; the host's admitted list and EmbeddedOpenOptions are the policy. The trust mode is the typed option EmbeddedOpenOptions::project_config_trust_mode (default LogOnly; Enforce refuses every edit lane with MCP's refusal text).
  • Routed batch_rename impact footer. A routed rename reads its impact footer from the routed worktree's post-image index; MCP reads the bound index after reindexing the written files. They agree whenever the untouched files of the two worktrees agree.

Two further limits come from libgit2 itself and are recorded in the description of PR #780, with one test kept ignored for them:

  • Global git config include. A malformed include in the user's global git config still refuses git preparation: libgit2 loads global config in every public open.
  • System gitattributes. System gitattributes still apply: libgit2 reads system attributes unconditionally.

Snapshot restore

Snapshot restore is engine-internal; you do not call a loader. A snapshot on disk may speed up re-proof but never confers authority by itself. The snapshot format is 10 as of 11.7.0, and a snapshot from an older format is an untrusted seed: it never restores. A copy is kept under .symforge/v11/quarantine/index-snapshots/, and the first snapshot write moves the original out of index.bin into .symforge/quarantine/index-snapshots/, so a downgrade must restore one of those copies. See Runtime-Model for the lifecycle.

Server integrators: server_api

With the default server feature, symforge::server_api::run(args: Vec<OsString>) -> Result<ServerExit, ServerBootstrapError> runs the whole SymForge CLI (argv[0] included). The symforge binary is a shim over it.

  • ServerExit is Success or RefusedToStart.
  • ServerBootstrapError is opaque: no public fields, so it cannot be built or matched exhaustively. Display gives the full cause chain.

Semver contract

symforge::embed is semver-public. A breaking change to any name or signature in it is a MAJOR bump; additive changes are MINOR. This is enforced at compile time: the contract test module in src/embed.rs names every contracted type and binds acquire, open_embedded_source, current_worktree, engine_info, index_census, index_progress and search_knowledge to their full signatures as fn pointers. A rename, removal or signature change fails SymForge's own embed test suite before a downstream integrator sees it. server_api carries its own shape test in src/server_api.rs.

symforge::embed::parity is versioned by API_VERSION, not by that contract module.

Changelog and migration guide

  • CHANGELOG.md opens with a hand-maintained Embedder API (embed) changes section, above the release entries. It tracks only changes to this facade.
  • docs/migrations/v11-index-lifecycle.md is the V10 to V11 guide: removed exports, the replacement atoms, snapshot and rollback constraints, and a compile-fix crib sheet.

Clone this wiki locally