Skip to content

Core Concepts Memory and Semantic Search System Memory Store Architecture

Jump to bottom
github-actions[bot] edited this page Aug 3, 2026 · 3 revisions

Memory Store Architecture

Referenced Files in This Document

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document explains the memory store architecture, focusing on the core interface, initialization process, data access patterns, lifecycle and connection management, error handling strategies, and the abstraction layer that unifies operations across storage backends. It also covers configuration examples, connection pooling, transaction-like semantics, performance considerations, caching mechanisms, scalability patterns, integration with the broader system, and concurrent access handling.

Project Structure

The memory store is implemented as a layered system:

  • High-level memory store API and orchestration live under services/memory.
  • The Qdrant-backed implementation lives under services/qdrant.
  • Shared utilities for concurrency, logging, validation, and tenant context are under utils.
  • Configuration and bootstrap glue connect the server to the memory store.
graph TB
subgraph "Memory Layer"
MS["store.ts"]
SI["store-init.ts"]
SM["store-methods.ts"]
SA["store-adapter.ts"]
SART["store-adapter-helpers.ts"]
SADH["store-adapter-default-handler.ts"]
SAHH["store-adapter-header-handler.ts"]
SATS["store-title-similarity-search.ts"]
SABF["activation-search-backfill.ts"]
SAF["activation-search-fields.ts"]
SAPM["qdrant-point-to-memory.ts"]
SVPS["validate-protocol-structure.ts"]
SVAM["validate-adapter-markdown-size.ts"]
end
subgraph "Qdrant Backend"
QSvc["service.ts"]
QConn["connection.ts"]
QIdx["index.ts"]
QInit["initialization.ts"]
QSearch["search.ts"]
QMemStore["memory-store.ts"]
QUpdates["memory-updates.ts"]
QRes["resources.ts"]
QSnap["snapshots.ts"]
QTypes["types.ts"]
QUtils["utils.ts"]
QUndi["undici-compat.ts"]
QQual["quality.ts"]
QRew["reward-propagation.ts"]
QList["listing.ts"]
QRet["memory-retrieval.ts"]
QProto["protocol.ts"]
end
subgraph "Shared"
KVFactory["key-value-store-factory.ts"]
KV["key-value-store.ts"]
RedisCache["redis-cache.ts"]
Redis["redis.ts"]
CCL["concurrency-limit.ts"]
GEH["global-error-handlers.ts"]
Log["log-core.ts"]
StructLog["structured-logger.ts"]
Tenant["tenant-context.ts"]
URI["uri-builder.ts"]
VComp["version-compare.ts"]
ZodJS["zod-to-jsonschema.ts"]
QColl["qdrant-collection-utils.ts"]
QQuery["qdrant-query-utils.ts"]
QVecMgmt["qdrant-vector-management.ts"]
QVecTypes["qdrant-vector-types.ts"]
end
MS --> SA
SA --> QSvc
SI --> MS
SM --> MS
SATS --> MS
SABF --> MS
SAF --> MS
SAPM --> MS
SVPS --> MS
SVAM --> MS
QSvc --> QConn
QSvc --> QInit
QSvc --> QSearch
QSvc --> QMemStore
QSvc --> QUpdates
QSvc --> QRes
QSvc --> QSnap
QSvc --> QList
QSvc --> QRet
QSvc --> QProto
QSvc --> QUtils
QSvc --> QTypes
QSvc --> QUndi
QSvc --> QQual
QSvc --> QRew
KVFactory --> KV
KVFactory --> RedisCache
RedisCache --> Redis
MS --> KVFactory
MS --> CCL
MS --> GEH
MS --> Log
MS --> StructLog
MS --> Tenant
MS --> URI
MS --> VComp
MS --> ZodJS
MS --> QColl
MS --> QQuery
MS --> QVecMgmt
MS --> QVecTypes
Loading

Diagram sources

Section sources

Core Components

  • Memory Store Interface and Orchestration: The central memory store module exposes high-level methods for training, activation, search, listing, and resource operations. It coordinates adapters, validators, and backend calls.
  • Adapter Abstraction: A consistent adapter contract allows different storage backends (e.g., Qdrant) to be plugged in uniformly. Default and header handlers implement common behaviors.
  • Initialization and Lifecycle: Dedicated initialization routines set up collections, indexes, and preconditions before serving requests.
  • Data Access Patterns: Methods encapsulate read/write flows, including vector indexing, metadata updates, and retrieval pipelines.
  • Validation Utilities: Protocol structure and markdown size validations ensure data integrity before persistence.
  • Search Enhancements: Title similarity search and activation search fields/backfills improve relevance and discoverability.

Key responsibilities by file:

Section sources

Architecture Overview

The memory store sits between application layers and the Qdrant vector database. It provides a unified API for CRUD, search, and artifact operations while abstracting backend specifics via an adapter pattern.

sequenceDiagram
participant App as "Application"
participant Mem as "Memory Store"
participant Adp as "Adapter"
participant QSvc as "Qdrant Service"
participant Conn as "Connection Pool"
participant Cache as "KV/Redis Cache"
App->>Mem : "train(space, protocol)"
Mem->>Mem : "validate inputs"
Mem->>Adp : "persistArtifact + index vectors"
Adp->>QSvc : "upsert points"
QSvc->>Conn : "acquire connection"
Conn-->>QSvc : "connection handle"
QSvc-->>Adp : "ack"
Adp-->>Mem : "result"
Mem->>Cache : "invalidate/update cache keys"
Mem-->>App : "success"
App->>Mem : "activate(space, slug)"
Mem->>Adp : "retrieve + score"
Adp->>QSvc : "search by title + filters"
QSvc->>Conn : "acquire connection"
Conn-->>QSvc : "connection handle"
QSvc-->>Adp : "results"
Adp-->>Mem : "ranked results"
Mem->>Cache : "cache result if applicable"
Mem-->>App : "activation payload"
Loading

Diagram sources

Detailed Component Analysis

Memory Store Interface and Orchestration

classDiagram
class MemoryStore {
+train(space, protocol) Promise
+activate(space, slug) Promise
+search(params) Promise
+list(params) Promise
+getArtifact(id) Promise
+updateArtifact(id, patch) Promise
+deleteArtifact(id) Promise
-validateInputs() void
-callAdapter(op, payload) Promise
-handleCache(key, fn) Promise
}
class StoreAdapter {
<<interface>>
+persistArtifact(data) Promise
+indexVectors(points) Promise
+searchByTitle(query, filters) Promise
+getResource(id) Promise
+updateResource(id, patch) Promise
+deleteResource(id) Promise
}
class QdrantService {
+upsertPoints(collection, points) Promise
+searchPoints(query, filters) Promise
+getPoint(id) Promise
+updatePoint(id, patch) Promise
+deletePoint(id) Promise
+listCollections() Promise
}
MemoryStore --> StoreAdapter : "uses"
StoreAdapter <|.. QdrantService : "implements"
Loading

Diagram sources

Section sources

Initialization Process and Lifecycle

flowchart TD
Start(["Startup"]) --> CheckCfg["Load configuration"]
CheckCfg --> InitQdrant["Initialize Qdrant service"]
InitQdrant --> EnsureCollections["Ensure collections exist"]
EnsureCollections --> ConfigureIndexes["Configure indexes and settings"]
ConfigureIndexes --> HealthCheck["Run health checks"]
HealthCheck --> Ready(["Ready to serve"])
Loading

Diagram sources

Section sources

Data Access Patterns and Abstraction Layer

classDiagram
class StoreAdapter {
<<interface>>
+persistArtifact(data) Promise
+indexVectors(points) Promise
+searchByTitle(query, filters) Promise
+getResource(id) Promise
+updateResource(id, patch) Promise
+deleteResource(id) Promise
}
class DefaultHandler {
+applyDefaults(payload) Promise
+normalizeResponse(result) Promise
}
class HeaderHandler {
+enrichWithHeaders(record) Promise
+stripSensitiveHeaders(record) Promise
}
class PointMapper {
+toDomain(point) Promise
+fromDomain(domain) Promise
}
StoreAdapter <|-- DefaultHandler : "extends"
StoreAdapter <|-- HeaderHandler : "extends"
StoreAdapter --> PointMapper : "uses"
Loading

Diagram sources

Section sources

Connection Management and Concurrency

sequenceDiagram
participant Caller as "Caller"
participant Limiter as "ConcurrencyLimiter"
participant ConnPool as "Qdrant Connection Pool"
participant Client as "Qdrant Client"
Caller->>Limiter : "request operation"
Limiter->>Limiter : "check capacity"
Limiter->>ConnPool : "acquire connection"
ConnPool-->>Limiter : "connection handle"
Limiter->>Client : "execute request"
Client-->>Limiter : "response"
Limiter->>ConnPool : "release connection"
Limiter-->>Caller : "result"
Loading

Diagram sources

Section sources

Error Handling Strategies

flowchart TD
Entry(["Operation Entry"]) --> TryOp["Try operation"]
TryOp --> Ok{"Success?"}
Ok --> |Yes| ReturnOk["Return normalized result"]
Ok --> |No| CatchErr["Catch error"]
CatchErr --> LogErr["Log structured error"]
LogErr --> Classify["Classify error type"]
Classify --> Transform["Transform to API error"]
Transform --> ReturnErr["Return error to caller"]
Loading

Diagram sources

Section sources

Transaction-Like Semantics and Atomicity

sequenceDiagram
participant Caller as "Caller"
participant Mem as "Memory Store"
participant Adp as "Adapter"
participant QSvc as "Qdrant Service"
Caller->>Mem : "beginTransaction()"
Mem->>Adp : "start batch"
Adp->>QSvc : "batch upsert points"
QSvc-->>Adp : "partial ack"
Adp->>QSvc : "commit batch"
QSvc-->>Adp : "commit result"
Adp-->>Mem : "transaction complete"
Mem-->>Caller : "success"
Loading

Diagram sources

Section sources

Caching Mechanisms and Invalidation

classDiagram
class KeyValueStoreFactory {
+create(config) KeyValueStore
}
class KeyValueStore {
<<interface>>
+get(key) Promise
+set(key, value, ttl) Promise
+del(key) Promise
}
class RedisCache {
+get(key) Promise
+set(key, value, ttl) Promise
+del(key) Promise
}
KeyValueStoreFactory --> KeyValueStore : "creates"
KeyValueStore <|.. RedisCache : "implements"
Loading

Diagram sources

Section sources

Search and Retrieval Pipelines

flowchart TD
Input(["Search Request"]) --> BuildQuery["Build query and filters"]
BuildQuery --> ExecuteSearch["Execute vector search"]
ExecuteSearch --> ScoreAndRank["Score and rank results"]
ScoreAndRank --> MapToDomain["Map points to domain objects"]
MapToDomain --> ApplyBackfill["Apply activation backfill if needed"]
ApplyBackfill --> ReturnResults["Return results"]
Loading

Diagram sources

Section sources

Validation and Data Integrity

flowchart TD
Start(["Input Payload"]) --> ValidateProtocol["Validate protocol structure"]
ValidateProtocol --> ValidP{"Valid?"}
ValidP --> |No| RejectP["Reject with error"]
ValidP --> |Yes| ValidateSize["Validate markdown size"]
ValidateSize --> ValidS{"Within limits?"}
ValidS --> |No| RejectS["Reject with error"]
ValidS --> |Yes| Proceed["Proceed to persistence"]
Loading

Diagram sources

Section sources

Artifact Operations

sequenceDiagram
participant Caller as "Caller"
participant Mem as "Memory Store"
participant Art as "Artifact Ops"
participant QSvc as "Qdrant Service"
Caller->>Mem : "getArtifact(id)"
Mem->>Art : "resolve artifact path"
Art->>QSvc : "read resource"
QSvc-->>Art : "artifact bytes"
Art-->>Mem : "artifact content"
Mem-->>Caller : "artifact response"
Loading

Diagram sources

Section sources

Integration with Broader System

graph TB
Server["HTTP Server"] --> Bootstrap["Bootstrap"]
Bootstrap --> MemStore["Memory Store"]
MemStore --> Tenant["Tenant Context"]
MemStore --> URI["URI Builder"]
MemStore --> Metrics["Metrics & Health"]
Loading

Diagram sources

Section sources

Dependency Analysis

The memory store depends on:

  • Qdrant service for vector operations and persistence.
  • Key-value store factory for caching.
  • Concurrency limiter for safe parallelism.
  • Logging and error handling utilities.
  • Tenant context and URI builder for multi-tenancy and addressing.
graph LR
Mem["Memory Store"] --> QSvc["Qdrant Service"]
Mem --> KV["KeyValueStoreFactory"]
Mem --> CCL["ConcurrencyLimiter"]
Mem --> Log["Logger"]
Mem --> Err["Error Handlers"]
Mem --> Tenant["TenantContext"]
Mem --> URI["URIBuilder"]
Loading

Diagram sources

Section sources

Performance Considerations

  • Vector Indexing: Use appropriate dimensionality and distance metrics; leverage collection-level settings for optimal recall vs. speed.
  • Batch Writes: Prefer batch upserts to reduce round-trips and increase throughput.
  • Concurrency Limits: Tune concurrency limits based on Qdrant capacity and network latency.
  • Caching: Cache frequent reads with short TTLs; invalidate aggressively on writes.
  • Query Optimization: Filter early and minimize payload sizes; use title similarity only when necessary.
  • Resource Monitoring: Monitor Qdrant metrics and adjust shard counts or replica factors as needed.

[No sources needed since this section provides general guidance]

Troubleshooting Guide

Common issues and diagnostics:

  • Connection errors: Verify Qdrant connectivity and credentials; check connection pool exhaustion.
  • Validation failures: Inspect protocol structure and markdown size constraints.
  • Search anomalies: Review filters, title similarity parameters, and backfill status.
  • Cache misses: Confirm key naming and invalidation triggers.
  • Errors and logs: Use structured logs and global error handlers to trace failures.

Section sources

Conclusion

The memory store provides a robust, extensible abstraction over Qdrant with clear separation of concerns: high-level orchestration, adapter-based backend access, validation, caching, and lifecycle management. Its design supports concurrent access, scalable search, and consistent operations across backends, integrating seamlessly with the broader application through standardized interfaces and utilities.

[No sources needed since this section summarizes without analyzing specific files]

Appendices

Configuration Examples

  • Qdrant connection:
    • Base URL, authentication, timeout, retry policy.
    • Collection names and vector dimensions.
  • Caching:
    • Redis URL, TTL defaults, fallback to in-memory.
  • Concurrency:
    • Max parallel operations per tenant.
  • Tenancy:
    • Tenant ID propagation and scope enforcement.

[No sources needed since this section provides general guidance]

KAIROS MCP

Clone this wiki locally