Proposal: expose bounded memory_search and memory_get over the existing memory_item store

[1625567290]

Revision note: This proposal has been narrowed after review. It now covers only the smallest model-facing read path over the existing Host-owned memory_item store. Stateful Plugin Platform work, source-range recall, working projections, observation/reflection, indexes, background consolidation, and new governance UI are not prerequisites for this slice.

Summary

A user should not need to repeat a durable preference, project fact, or prior decision in every conversation.

Maka already extracts atomic items and stores them with keys, Scope, temporal state, lifecycle, and RuntimeEvent provenance. The missing product link is smaller: the Agent cannot search or read those accepted items.

The proposed first slice is therefore two bounded, Host-owned model tools over the existing store:

• memory_search — find active Items by exact or prefix key under the current Runtime-authorized Scope;
• memory_get — read one authorized Item by ID with its current version and provenance metadata.

No storage migration, Plugin migration, derived memory layer, vector dependency, or new background process is required.

Current implementation gap

At the reviewed main commit, Maka already has:

• atomic memory_item records with global and workspace Scope, temporal fields, lifecycle state, typed keys, and source event identities;
• MemoryItemStore.readItem(itemId);
• MemoryItemStore.searchByKeys({ terms, match, workspaceKey, includeArchived, limit });
• exact and prefix matching with deterministic normalization and bounded term/result counts;
• Runtime Policy fields memory.enabled and memory.agentReadEnabled, with Agent reads disabled by default;
• Incognito and extraction policy enforcement on the write path.

What it does not have is a model-facing tool wired to readItem or searchByKeys. The current prompt projection reads the separate local-memory document; it does not expose the accepted SQLite Item store to the Agent.

This proposal connects that existing read capability. It does not redesign the memory lifecycle.

Proposed first slice

1. Runtime Host owns authorization

Runtime Host derives the current Workspace identity and applies Runtime Policy before a tool is admitted or executed.

• Tool input must not accept workspaceKey, Profile ID, Session ID, or another caller-selected Scope.
• Search covers active Global Items plus active Items for the current Workspace.
• A Workspace-scoped Item must never be visible from another Workspace.
• Incognito, memory.enabled = false, or memory.agentReadEnabled = false makes the read tools unavailable or returns a policy-denied result without querying storage.
• Archived Items are excluded from Agent reads in this slice.

2. memory_search

Suggested input:

{
  terms: string[];
  match?: 'exact' | 'prefix';
  limit?: number;
}

V1 should preserve the current store semantics: match any distinct normalized term and rank by the number of matched terms. The model-facing limit should default to 5 and be capped at 10, below the storage ceiling.

Each result should contain only what is needed to choose or use an Item:

{
  itemId: string;
  version: number;
  content: string;
  kind: string;
  statementType: string;
  temporalType: string;
  eventStartedAt: number | null;
  eventEndedAt: number | null;
  observedAt: number;
  matchedKeys: string[];
  sourceCount: number;
}

The output must have a hard aggregate size limit and an explicit truncation indicator. It must not expose records from a broader Scope when the result budget is exhausted or a Scope identity is unavailable.

3. memory_get

Suggested input:

{ itemId: string }

The Host resolves the Item, checks that it is active and belongs to Global or the current Workspace Scope, then returns the current Item, keys, version, temporal state, and source identities. A missing, archived, or unauthorized Item should produce the same non-disclosing not-found result.

memory_get does not retrieve raw event content. Source-range recall is a separate capability that should be added only if evaluation shows that Item content and provenance identities are insufficient for trustworthy answers.

4. No implicit prompt injection

This slice adds explicit tools only. It does not automatically inject SQLite Items into every prompt and does not change the existing local-memory document projection.

That keeps cost and behavior observable: an Item enters model context only through a recorded tool call and bounded tool result.

Acceptance gates

Deterministic CI

• exact-key and prefix-key search return the expected active Items in deterministic order;
• Global Items are visible from any normal Workspace, while Workspace Items are visible only from their owning Workspace;
• Incognito and both Memory policy gates prevent storage reads;
• archived Items are absent from memory_search and indistinguishable from missing Items in memory_get;
• the model cannot supply or widen Scope identifiers;
• result count and aggregate output budgets are enforced with explicit truncation metadata;
• Item version, temporal state, matched keys, and provenance availability survive the tool projection;
• CJK two-character keys, mixed-language keys, code identifiers, duplicate terms, empty terms, and over-limit inputs are covered;
• one Run keeps the tool composition admitted at its start across a concurrent policy change.

Capability evaluation

Use fixture conversations in which a later Session asks about a durable preference, project fact, or prior decision established in an earlier Session.

Compare:

1. exact recent history only;
2. exact recent history plus memory_search/memory_get.

Report:

• continuity answer accuracy;
• preference and constraint adherence;
• unsupported-memory usage;
• correct abstention when no matching Item exists;
• cross-Scope leakage;
• retrieved Item precision;
• tool-result characters/tokens;
• added latency.

The first slice succeeds only if it improves later-session continuity without increasing unsupported claims or Scope leakage.

Explicit non-goals

The following are not prerequisites and are not part of this proposal:

• moving Memory into Plugin-owned state;
• completing the Stateful Plugin Platform;
• generic RuntimeEvent observation or Context contribution APIs;
• source-range event recall;
• Working Projection or automatic pinning;
• Observation or Reflection;
• lexical, vector, entity, or graph indexes;
• background consolidation;
• new governance UI;
• feedback, forgetting, or automatic cross-session injection;
• changing the existing extraction pipeline or SQLite Schema.

What would justify another layer

Additional mechanisms should be proposed only against a measured failure:

• add source-range recall if unsupported answers or trust failures require original evidence;
• add a small Working Projection if repeatedly retrieving ubiquitous preferences creates measurable cost or misses;
• add Observation/Reflection if atomic Item recall fails on episode-level questions;
• add lexical or vector indexing if exact/prefix key search has inadequate measured coverage;
• consider a Stateful Plugin boundary separately if there is a platform ownership or deployment objective independent of this read-path slice.

Questions for maintainers

1. Is a Host-owned memory_search/memory_get pair the right smallest next slice under Title: feat(memory): add an automatic long-term memory lifecycle #1615?
2. Should memory.agentReadEnabled gate both tools, matching the existing Agent-read authority?
3. Are default 5 / maximum 10 search results appropriately bounded for V1?
4. Should the tool projection include source event identities, or only sourceCount until a source-range design exists?
5. If this direction is accepted, should implementation continue as the recall slice of Title: feat(memory): add an automatic long-term memory lifecycle #1615 rather than as a separate lifecycle project?

Relationship to existing work

• Title: feat(memory): add an automatic long-term memory lifecycle #1615 remains the authority for the automatic long-term-memory lifecycle. This proposal narrows only its missing model-facing recall slice.
• Proposal: Evolve Local Memory — from Full Injection to Extraction + On-Demand Retrieval + Layered Injection #2335 was already consolidated into Title: feat(memory): add an automatic long-term memory lifecycle #1615 and should not be reopened as a separate retrieval architecture.
• Proposal: lifecycle-managed Extension Runtime for safe agent self-extension #2973 and feat(runtime): add Plugin Platform foundation #3729 concern the Extension/Plugin Platform. They are not prerequisites for this slice.
• feat(memory): use stable workspace identities #2986 remains separate Workspace-identity work and is not taken over here.

References

• Reviewed Maka main commit: 3db83c2
• Item and search contracts:

  maka/packages/core/src/long-term-memory.ts

  Lines 75 to 111 in 3db83c2

  	export interface MemoryItem {
  	readonly itemId: string;
  	readonly version: number;
  	readonly content: string;
  	readonly kind: MemoryItemKind;
  	readonly statementType: MemoryStatementType;
  	readonly temporalType: MemoryTemporalType;
  	readonly scopeType: MemoryScopeType;
  	readonly scopeKey: string | null;
  	readonly eventStartedAt: number | null;
  	readonly eventEndedAt: number | null;
  	readonly observedAt: number;
  	readonly lifecycleState: MemoryLifecycleState;
  	/** How the current content was produced, not how this stable Item was first created. */
  	readonly origin: MemoryItemOrigin;
  	readonly contentHash: string;
  	readonly createdAt: number;
  	readonly updatedAt: number;
  	}
  	export interface MemoryItemKeyInput {
  	readonly key: string;
  	readonly keyType: MemoryKeyType;
  	readonly keyOrigin: MemoryKeyOrigin;
  	}
  	export interface MemoryItemKey extends MemoryItemKeyInput {
  	readonly normalizedKey: string;
  	}
  	/** Durable provenance for the current Item content. */
  	export interface MemoryItemSource {
  	readonly sessionId: string;
  	readonly runId: string;
  	readonly turnId: string;
  	readonly eventId: string;
  	}

• Store read interface:

  maka/packages/core/src/long-term-memory.ts

  Lines 295 to 332 in 3db83c2

  	export interface MemoryItemRecord {
  	readonly item: MemoryItem;
  	readonly keys: readonly MemoryItemKey[];
  	readonly sources: readonly MemoryItemSource[];
  	}
  	export interface SearchMemoryItemsByKeyRequest {
  	/** Match any distinct term and rank Items by the number of matched terms. */
  	readonly terms: readonly string[];
  	readonly match: 'exact' | 'prefix';
  	/** Omit to search global Items only; provide to include global plus this workspace. */
  	readonly workspaceKey?: string;
  	readonly includeArchived?: boolean;
  	readonly limit?: number;
  	}
  	export interface MemoryItemStore {
  	applyMutations(request: ApplyMemoryMutationsRequest): Promise<MemoryWriteOperationResult>;
  	commitExtraction(request: CommitMemoryExtractionRequest): Promise<MemoryExtractionCommitResult>;
  	initializeExtractionCursor(
  	sessionId: string,
  	processedOrdinal: number,
  	): Promise<MemoryExtractionCursor>;
  	readExtractionCursor(sessionId: string): Promise<MemoryExtractionCursor | undefined>;
  	readPendingExtractionFailure(
  	sessionId: string,
  	): Promise<PendingMemoryExtractionFailure | undefined>;
  	recordCompactionPolicyDenial(
  	denial: MemoryCompactionPolicyDenial,
  	): Promise<MemoryCompactionPolicyDenial>;
  	readCompactionPolicyDenials(sessionId: string): Promise<readonly MemoryCompactionPolicyDenial[]>;
  	settleExtractionFailure(
  	request: SettleMemoryExtractionFailureRequest,
  	): Promise<SettleMemoryExtractionFailureResult>;
  	readExtractionReceipt(operationId: string): Promise<MemoryExtractionReceipt | undefined>;
  	readItem(itemId: string): Promise<MemoryItemRecord | undefined>;
  	searchByKeys(request: SearchMemoryItemsByKeyRequest): Promise<readonly MemoryItemRecord[]>;
  	readOperation(operationId: string): Promise<MemoryWriteOperationResult | undefined>;

• SQLite read implementation:

  maka/packages/storage/src/sqlite-long-term-memory-store.ts

  Lines 842 to 885 in 3db83c2

  	async readItem(itemId: string): Promise<MemoryItemRecord | undefined> {
  	this.#assertOpen();
  	return this.#readSnapshot(() => this.#readItemRecord(normalizeIdentifier(itemId, 'itemId')));
  	}
  	async searchByKeys(request: SearchMemoryItemsByKeyRequest): Promise<readonly MemoryItemRecord[]> {
  	this.#assertOpen();
  	if (request.match !== 'exact' && request.match !== 'prefix') {
  	throw new Error('Memory key match must be exact or prefix');
  	}
  	if (!Array.isArray(request.terms) || request.terms.length === 0) {
  	throw new Error('Memory key search requires at least one term');
  	}
  	if (request.terms.length > MAX_SEARCH_TERMS) {
  	throw new Error(`Memory key search accepts at most ${MAX_SEARCH_TERMS} terms`);
  	}
  	if (request.includeArchived !== undefined && typeof request.includeArchived !== 'boolean') {
  	throw new Error('Memory key includeArchived must be a boolean');
  	}
  	const terms = [...new Set(request.terms.map(normalizeSearchTerm))].sort(compareText);
  	const workspaceKey =
  	request.workspaceKey === undefined
  	? undefined
  	: normalizeIdentifier(request.workspaceKey, 'workspaceKey');
  	const limit = request.limit ?? 20;
  	if (!Number.isSafeInteger(limit) || limit < 1 || limit > MAX_SEARCH_RESULTS) {
  	throw new Error(`Memory key search limit must be between 1 and ${MAX_SEARCH_RESULTS}`);
  	}
  	const query = buildSqliteMemoryKeySearchQuery({
  	terms,
  	match: request.match,
  	workspaceKey,
  	includeArchived: request.includeArchived ?? false,
  	limit,
  	});
  	return this.#readSnapshot(() => {
  	const rows = this.#database.prepare(query.sql).all(...query.parameters) as Array<{
  	item_id?: unknown;
  	}>;
  	return rows.map((row) => {
  	if (typeof row.item_id !== 'string') throw new Error('Invalid Memory Item search result');
  	const record = this.#readItemRecord(row.item_id);

• Runtime Memory policy:

  maka/packages/core/src/runtime-policy.ts

  Lines 128 to 130 in 3db83c2

  	readonly memory: {
  	readonly enabled: boolean;
  	readonly agentReadEnabled: boolean;

• Long-term-memory lifecycle: Title: feat(memory): add an automatic long-term memory lifecycle #1615

[likun666661]

I think this proposal starts from the wrong question.

The user problem is simple: a user should not need to repeat durable preferences, project facts, and prior decisions across conversations. The current implementation gap is also concrete: Maka already has RuntimeEvent evidence, automatic extraction, the SQLite memory_item store, source references, scopes, cursors, receipts, and exact/prefix key search, but the Agent still has no model-facing read path over that store.

The shortest falsifiable path is therefore:

conversation evidence
  -> extracted atomic item
  -> scoped durable store
  -> bounded search/get when a later turn needs it
  -> better continuity for the user

This proposal replaces that question with a much larger one: what generic platform capabilities must exist before a perfect stateful Memory Plugin can be built? That conflates three separate projects:

1. proving useful long-term-memory behavior;
2. completing the Stateful Plugin Platform;
3. experimenting with observation, reflection, layered context, indexes, background consolidation, and governance projections.

The Plugin boundary is a deployment and ownership decision. It is not evidence that Tool contribution, generic Plugin state, RuntimeEvent observation, Context contribution, background work, and a new governance projection must all precede useful memory. Requiring all six turns a missing read path into a platform program.

The three-plane model is a reasonable description of authority, but it does not justify three engineering subsystems. For the first useful slice, RuntimeEvent and memory_item are enough. Search results are already context. Observation groups, reflections, working projections, context packs, optional indexes, projection health, and replacement machinery should each be introduced only after a measured failure shows that direct item recall is insufficient.

I would reset the proposal to the following sequence:

1. Keep the existing Host-owned memory_item store.
2. Expose bounded, policy-checked memory_search and memory_get over the current Scope.
3. Run Shadow and semantic evaluation against exact recent history.
4. Add source-range recall only if unsupported answers or trust failures require it.
5. Add a small working projection only if repeated retrieval of ubiquitous preferences is measurably poor.
6. Add observation/reflection only if atomic-item recall demonstrably fails on episode-level questions.

Scope isolation, Incognito, bounded results, provenance, and fail-closed behavior are necessary safeguards and should remain. The rest has not yet earned its complexity.

If the real objective is to use Memory as the forcing function for a Stateful Plugin Platform, that is a legitimate but different proposal. It should be named and justified as a platform investment, with Memory as its validation consumer. It should not present the full Extension Runtime roadmap as if it were a prerequisite derived from the memory product problem.

So I do not think the current prerequisite framing should be accepted. First connect the read side that already exists in storage, measure what it solves, and let concrete failures pay for each additional layer.

[1625567290]

You're right. I conflated the missing product read path with a separate platform investment.

I have narrowed the title and body to the existing Host-owned memory_item store plus bounded memory_search/memory_get, gated by the current Scope, Incognito, and memory.agentReadEnabled. Plugin state, generic RuntimeEvent/Context contribution APIs, source-range recall, working projections, observation/reflection, indexes, background consolidation, and new governance UI are now explicit non-goals.

The revised evaluation compares exact recent history alone against recent history plus direct Item search/get. Any additional layer now has to be justified by a measured failure. Thanks for making that boundary explicit.