Skip to content

20 Token Efficient Retrieval

Heinrich Krupp edited this page Nov 5, 2026 · 3 revisions

Token-Efficient Retrieval

Reviewed for v12.6.0 on 2026-11-05. Source: maintained repository guide.

How to stop memory_search from flooding an agent's context, and how to use the two-phase memory_explore / memory_detail pair.

Three independent levers:

  1. Bounding a search response — works on every backend, no setup. Start here.
  2. Query-aware summaries — opt-in, requires a configured LLM provider.
  3. The two-phase knowledge map — needs a populated entity graph, and a graph-capable backend.

1. Bounding a search response

An unbounded memory_search on a large store routinely returns 10k+ tokens, most of it content the caller will not read. Two parameters control that:

Parameter Default Effect
limit 10 Maximum number of memories returned
max_response_chars unlimited Caps total response size, truncating at memory boundaries so no record is cut mid-content
{"query": "replication message bus consistency", "limit": 5, "max_response_chars": 30000}

30000-50000 is the range to start with. Truncation reports what was dropped, so the caller can narrow the query or raise the cap deliberately rather than silently losing results.

If you are calling the service from code rather than as an MCP tool, the code-execution interface has compact result types (CompactMemory, CompactSearchResult) that cut 85-91% of the tokens of full Memory objects — see docs/api/code-execution-interface.md.

2. Query-aware search summaries

Use summarize: true when you want an answer scoped to the current query rather than every retrieved memory verbatim:

{"query": "replication message bus consistency", "limit": 10, "summarize": true}

The caller's flag defaults to false. The server operator must also explicitly set MCP_SEARCH_SUMMARIZE_ENABLED=true and restart the server. This setting is off by default, independently of Harvest and provider configuration; unset, empty, or invalid values leave it disabled. When disabled, summarize: true returns raw results with a Summarization unavailable: disabled by MCP_SEARCH_SUMMARIZE_ENABLED warning and makes no summarization provider call.

Retrieval, filters, search fallback, and retrieval plugins run first. Summarization then uses the existing Harvest LLM provider chain; it does not require Harvest to be enabled. For example, a local OpenAI-compatible endpoint can be configured with:

MCP_SEARCH_SUMMARIZE_ENABLED=true
HARVEST_LLM_PROVIDERS=local
HARVEST_LLM_LOCAL_BASE_URL=http://localhost:11434/v1
HARVEST_LLM_LOCAL_MODEL=your-installed-model

Add HARVEST_LLM_LOCAL_API_KEY if the endpoint requires authentication. The legacy GROQ_API_KEY configuration also works, but does not enable search summarization on its own. Operator permission applies to local and remote clients, including read-scope MCP clients. They can then request summaries that send the query and selected memory records to the configured providers and consume provider quota. Use a provider chain containing only local endpoints to keep this data on-host.

The complete input prompt is capped at 12,000 characters, including the query, instructions, and metadata. Records that do not fit are omitted whole, and later smaller records may still fit. The existing provider output budget is 200 tokens. Generated text exceeding 2,000 characters is rejected even if a provider ignores that token budget. Non-finite metadata values (NaN or infinity) also fail the summary path so the response always contains valid JSON. Each provider uses a 10-second HTTP timeout, with a 30-second deadline for the whole summarization call. These are fixed MVP defaults; summarize accepts a boolean, not a budget object.

A successful MCP text response contains JSON:

{
  "summarized": true,
  "summary": "Acknowledgements preceded replicated state propagation [1].",
  "source_hashes": ["<hash-a>"],
  "snapshot": [
    {"content_hash": "<hash-a>", "tags": ["replication"], "created_at_iso": "2026-09-01T10:00:00Z"}
  ],
  "summarized_count": 1,
  "omitted_count": 0,
  "total": 1,
  "query": "replication message bus consistency",
  "mode": "semantic",
  "provider": "local",
  "model": "your-installed-model"
}

snapshot is the pre-summary keep-set: a deep copy of every non-content field in each record sent to the model, in citation order ([1] refers to its first entry). It includes available source metadata, not only the abbreviated fields above. The internal access_queries history is excluded from both the provider input and snapshot, whether flattened into the record or nested under metadata. Stored access history is left intact. source_hashes lists the cited hashes in order of first use; the snapshot also retains sources the model did not cite. omitted_count reports records excluded by the input budget. Debug details and requested derived beliefs are included under debug and beliefs when available.

Source metadata is kept outside the generated answer. Missing hashes, tags, or creation timestamps, duplicate hashes, empty output, missing citations, and invalid or incomplete citations fail the summary path. Provider failures, missing configuration, and searches without a query also return normal raw results with a Summarization unavailable warning. Empty searches do not call the model. If the complete summary response cannot fit max_response_chars, the tool falls back rather than cutting source metadata. A positive cap bounds the entire summary fallback response, including warnings, headers, and requested beliefs. Raw results take priority: the longest prefix of complete records that fits is returned, and requested beliefs are included only if the remaining space allows. An optional-section omission notice is added when it fits. Raw memory records are omitted whole when they do not fit; very small caps may return only the beginning of the warning. Searches without summarization retain their existing response limiting behavior.

The keep-set is request-local, not a persistent audit or rollback copy. Original memory content is never rewritten or deleted. Retrieve an original by hash through GET /api/memories/{content_hash}, or narrow a subsequent memory_search with summarize: false to inspect raw content. Citation checks verify source identity; they cannot guarantee that every generated statement correctly represents its source. Automatic threshold triggering and stateful follow-up conversations are outside this first version.

3. The two-phase knowledge map

memory_explore returns a map of entities with a few ranked chunks each; memory_detail then returns the full ranked chunk list for one entity. The point is to let a caller decide where to drill in before pulling content into context.

{"query": "authentication design", "max_entities": 5, "chunks_per_entity": 2}
{"entity_id": "authentication-design", "limit": 20, "include_related": false}

memory_explore parameters: max_entities (default 10), chunks_per_entity (default 3), tags, min_score, store, and scoring: "composite" to re-rank by relevance plus graph proximity and entity centrality. memory_detail takes the entity_id from the map, plus limit (default 50, max 200), include_related (default true) and max_hops (default 1).

Prerequisite: the entity graph must be populated

memory_explore returns {"entities": [], "count": 0} for every query until entities exist. Nothing populates them by default. Check first:

{"action": "list_entities", "limit": 10}

via memory_graph. If that returns {"entities": [], "total": 0}, the two-phase tools have nothing to work with, and the fix is one of:

  • Batch, for memories you already have — memory_quality with {"action": "rebuild_entities"} previews a full entity-graph rebuild. Review the report, then use {"action": "rebuild_entities", "dry_run": false} to apply it. See Rebuilding the entity graph before applying. maintain can also populate has_entity, but does not create shares_entity.
  • Ongoing, for memories stored from now on — set MCP_ENTITY_LINKING_ENABLED=1. This links at store time and does not touch existing memories.
  • A single memory — memory_graph with {"action": "extract_entities", "hash": "<content hash>"}.

Graph storage exists for the sqlite_vec, hybrid and milvus backends. Cloudflare has none, so these tools are unavailable there.

The scan limit will surprise you on a large store

maintain scans at most MCP_MAINTAIN_SCAN_LIMIT memories per run, default 2000, and it takes the first N in the order the storage returns them. Running it repeatedly re-scans the same 2000 — it does not walk forward through the store. On any store larger than the limit, everything past the first N is never reached, however many times you run it.

Set MCP_MAINTAIN_SCAN_LIMIT=0 for unlimited, and expect the run to take proportionally longer:

MCP_MAINTAIN_SCAN_LIMIT=0 memory restart

What counts as an entity

Extraction is deliberately pattern-based and high-precision, not semantic NER. It finds:

  • @mentions (typed as person)
  • #hashtags (typed as tag)
  • URLs
  • file paths
  • the memory's tags
  • any terms listed in MCP_ENTITY_CUSTOM_TERMS (comma-separated), matched on word boundaries

Every tag becomes an entity by default, so a generic tag such as bugfix or project:<name> links every memory that carries it. Two optional settings narrow that, each a comma-separated list of fnmatch patterns matched case-insensitively:

  • MCP_ENTITY_TAG_INCLUDE — only matching tags become entities;
  • MCP_ENTITY_TAG_EXCLUDE — matching tags never do.

When both are set, include is applied first, then exclude. Unset keeps every tag. Entities found in the content (#hashtags, @mentions, URLs, paths) are not filtered. The filter applies to extraction from then on; use rebuild_entities to update existing edges.

MCP_ENTITY_TAG_INCLUDE=task:*

It does not recognise "Postgres", "Kafka" or "OAuth" as entities just because they appear in prose. On a store whose memories are well tagged but whose content carries none of the patterns above, tags do most of the work — which is why tags being dropped (#218) left graphs empty.

For domain-specific extraction, MCP_ENTITY_EXTRACTOR_MODULES accepts comma-separated module.path:ClassName entries implementing the DomainExtractor protocol.

Rebuilding the entity graph

After changing entity filters or custom terms, correcting old extraction errors, or enabling linking on an existing store, preview a complete rebuild with memory_quality:

{"action": "rebuild_entities"}

This defaults to dry_run: true and changes no memories or edges. edges_before counts the rows that would be deleted, and edges_after counts the replacement rows. Both contain has_entity and shares_entity. A shares_entity pair uses two directed rows; shares_entity_pairs_before and shares_entity_pairs_after count pairs instead. memories_scanned counts all live memories across all named stores, including superseded versions, in creation order. MCP_MAINTAIN_SCAN_LIMIT does not apply. The graph is selected from the server's active storage, including Hybrid's primary, and a Milvus preview leaves a missing graph collection absent. A read failure stops the operation before deleting any edges. Rebuilds load configured domain extractors afresh and stop before deletion if a plugin fails to load or extract entities.

Back up the database first and pause other writers and consolidation jobs for the entire preview/apply window. Restart the server to load changed extraction settings, then review the preview and apply:

{"action": "rebuild_entities", "dry_run": false}

The operation deletes only has_entity and shares_entity edges, then recreates them using the current extractor and EntityLinker. Other relationships and their metadata and timestamps are preserved. If a pair already has a non-entity relationship, the rebuild skips a shares_entity pair that would overwrite it; protected_edges reports the number of non-entity rows retained. Memory contents, hashes, timestamps, tags, and store assignments are unchanged. Rebuilding explicitly does not require MCP_ENTITY_LINKING_ENABLED to be enabled; that flag controls future store-time linking.

The newest-first entity lookup window and linking cap are the same as at store time. Equal creation timestamps are ordered by content hash for repeatable results. The result reproduces linking with today's settings, rather than preserving historical entity edges. The old SQLite-only backfill script has been replaced by this action.

SQLite, Hybrid (its local graph), and Milvus are supported. Cloudflare reports an unsupported graph backend and does nothing. This is a write operation requiring OAuth write scope, even for a preview. Only a boolean dry_run: false applies changes.

The full memory scan, graph snapshot, and replacement plan are held in memory before any deletion. Large stores require proportionally more RAM, and Milvus performs an RPC per edge write, so a rebuild can take a while. Pause other writers: the process-local rebuild lock does not coordinate separate servers, and this is not a cross-backend transaction. A write failure returns success: false, errors and progress counters; the graph may be partially rebuilt. Fix the failure and rerun, or restore the backup.

What summary actually is

The summary field on each entity is the first sentence of up to three of its top chunks, each truncated to 150 characters. Chunk content itself is capped at 500 characters. It is snippet concatenation, useful for deciding where to drill in — not a summarization of the entity.

Query-scoped entity selection

memory_explore selects entities linked to the retrieved query chunks, starting with the highest-relevance chunks and stopping at max_entities. If none of those chunks has an entity link, it falls back to the graph's global entity list so partially populated graphs remain discoverable. A fallback entity receives an empty top_chunks list and summary unless one of its linked memories is present in the query results; unrelated query chunks are never attached to it.

Which to reach for

Situation Use
Search responses too large limit + max_response_chars
Want a concise answer to the current query memory_search with summarize: true (needs an LLM provider)
Want an overview before pulling content memory_explore → memory_detail
Restrict results to one known entity memory_search with entity (needs a populated graph)
Calling from Python rather than as an MCP tool code-execution interface, compact types

Clone this wiki locally