From 546b787d06709014603c9dacf8f788864a153bc4 Mon Sep 17 00:00:00 2001 From: mrorigo Date: Sun, 9 Nov 2025 15:34:35 +0100 Subject: [PATCH] docs: update developer guide with user-focused api, search, provider, and conscious mode docs --- .../consolidation-service-architecture.md | 72 ++- .../advanced-features/duplicate-management.md | 66 +-- docs/developer/api/core-api.md | 487 ++++++++++++++---- docs/developer/api/search-api.md | 401 +++++++++----- .../developer/architecture/system-overview.md | 52 +- .../core-concepts/memory-management.md | 10 +- .../core-concepts/search-strategies.md | 364 ++++++------- docs/developer/index.md | 11 +- .../conscious-mode-and-background-jobs.md | 253 +++++++++ docs/developer/providers/overview.md | 290 +++++++++++ 10 files changed, 1517 insertions(+), 489 deletions(-) create mode 100644 docs/developer/operations/conscious-mode-and-background-jobs.md create mode 100644 docs/developer/providers/overview.md diff --git a/docs/developer/advanced-features/consolidation-service-architecture.md b/docs/developer/advanced-features/consolidation-service-architecture.md index 662f318..90129df 100644 --- a/docs/developer/advanced-features/consolidation-service-architecture.md +++ b/docs/developer/advanced-features/consolidation-service-architecture.md @@ -30,51 +30,79 @@ const duplicates = await consolidationService.detectDuplicateMemories( 0.75 ); -if (duplicates.length > 0) { +if (duplicates.length > 1) { const primaryId = duplicates[0].id; const others = duplicates.slice(1).map(d => d.id); - const validation = await consolidationService.validateConsolidationEligibility(primaryId, others); - if (validation.isValid) { + // Example pattern; consult the actual ConsolidationService implementation + // for the exact method names available in this version. + if (consolidationService.validateConsolidationEligibility) { + const validation = await consolidationService.validateConsolidationEligibility(primaryId, others); + if (!validation.isValid) { + console.log('Consolidation not safe:', validation.reasons); + // Abort or adjust as needed + } + } + + if (consolidationService.previewConsolidation) { const preview = await consolidationService.previewConsolidation(primaryId, others); console.log(preview.summary); - - const result = await consolidationService.consolidateMemories(primaryId, others); - console.log(`Merged ${result.consolidated} memories`); } + + const result = await consolidationService.consolidateMemories(primaryId, others); + console.log(`Merged ${result.consolidated} memories`); } ``` -The service performs safety checks, records audit entries, and handles rollbacks if consolidation fails. +The service is responsible for safety checks and atomic updates. Exact helper methods +(e.g. eligibility validation, preview, analytics) depend on the concrete `ConsolidationService` +implementation; use the available methods on your version rather than assuming undocumented ones. ## Scheduling -`DatabaseManager` can run consolidation on a schedule. Use it for background maintenance jobs: +`DatabaseManager` can run consolidation on a schedule. These controls are advanced and should be +treated as internal; the stable entrypoint remains `memori.getConsolidationService()`. ```typescript -memori.getConsolidationService(); // ensure service initialised - -memori['dbManager'].startConsolidationScheduling({ - intervalMinutes: 120, - maxConsolidationsPerRun: 20, - similarityThreshold: 0.8, - dryRun: true -}); +import { Memori } from 'memorits'; + +const memori = new Memori({ databaseUrl: 'file:./memori.db' }); +await memori.enable(); + +const consolidationService = memori.getConsolidationService(); + +// For most applications, call consolidationService methods explicitly from your own scheduler. +// Example (simplified): + +async function runScheduledConsolidation() { + const duplicates = await consolidationService.detectDuplicateMemories('...', 0.8); + // apply your own selection/validation/merge policy here using the public service API +} ``` -The scheduling methods live on `DatabaseManager`. Access them from scripts where you control the lifecycle. Scheduled runs log progress under the `DatabaseManager` component. +If you reach into `DatabaseManager` to use `startConsolidationScheduling` or related methods, +treat that as unstable, internal-only usage: APIs and access patterns may change between versions. ## Metrics & Analytics +If your `ConsolidationService` implementation exposes analytics helpers +(e.g. `getConsolidationAnalytics`, `getConsolidationHistory`), you can use them to +inspect duplicate density, success rates, and historical operations: + ```typescript -const analytics = await consolidationService.getConsolidationAnalytics(); -console.log(`Success rate: ${analytics.successRate}%`); +if (consolidationService.getConsolidationAnalytics) { + const analytics = await consolidationService.getConsolidationAnalytics(); + console.log(`Success rate: ${analytics.successRate}%`); +} -const history = await consolidationService.getConsolidationHistory({ limit: 50 }); -console.log(`Recent consolidations: ${history.length}`); +if (consolidationService.getConsolidationHistory) { + const history = await consolidationService.getConsolidationHistory({ limit: 50 }); + console.log(`Recent consolidations: ${history.length}`); +} ``` -These APIs provide insight into how often duplicates appear, how long consolidations take, and which namespaces are most affected. +Always consult the concrete `ConsolidationService` type in your version of `memorits` +to see which analytics methods are available. ## Extending the Service diff --git a/docs/developer/advanced-features/duplicate-management.md b/docs/developer/advanced-features/duplicate-management.md index 02e0a4e..3802562 100644 --- a/docs/developer/advanced-features/duplicate-management.md +++ b/docs/developer/advanced-features/duplicate-management.md @@ -4,7 +4,7 @@ Memorits includes tooling to surface and manage duplicate memories so your knowl ## Finding Potential Duplicates -`Memori.findDuplicateMemories` wraps the duplicate detection pipeline. It compares supplied content against existing memories using Jaccard and trigram similarity metrics. +`Memori.findDuplicateMemories` wraps the duplicate detection pipeline. It compares supplied content against existing memories using the internal similarity scoring implemented by the duplicate management components (threshold-based, implementation detail), returning only candidates above the configured similarity threshold. ```typescript import { Memori } from 'memorits'; @@ -33,53 +33,53 @@ Behind the scenes `DuplicateManager`: - Filters results above the requested threshold. - Emits structured logs (`component: DuplicateManager`) so you can trace detection. -## Conscious Context Consolidation +## Consolidation via Public API -When conscious mode is enabled you can trigger duplicate clean-up through `ConsciousAgent`. `Memori` invokes it internally, but you can call it manually for batch jobs: +When you need structured consolidation workflows, use the consolidation service exposed through `Memori` instead of reaching into private agents: ```typescript -await memori.initializeConsciousContext(); - -const consolidation = await memori['consciousAgent']?.consolidateDuplicates({ - similarityThreshold: 0.8, - dryRun: true -}); +import { Memori } from 'memorits'; -if (consolidation) { - console.log(`Analysed ${consolidation.totalProcessed} memories, found ${consolidation.duplicatesFound} potential duplicates.`); -} -``` +const memori = new Memori({ databaseUrl: 'file:./memori.db' }); +await memori.enable(); -> **Note:** `consciousAgent` is a private property; use TypeScript index access as shown only in controlled scripts. A public wrapper is planned in the API surface. +const consolidationService = memori.getConsolidationService(); -The consolidation run: +const duplicates = await consolidationService.detectDuplicateMemories( + 'Reminder: invoices go out on the 5th.', + 0.75 +); -- Validates that the primary memory exists and is not already being consolidated. -- Estimates memory usage and captures basic performance statistics. -- Can run in `dryRun` mode to preview results. +if (duplicates.length > 1) { + const primaryId = duplicates[0].id; + const others = duplicates.slice(1).map(d => d.id); -## Supporting Utilities + const validation = await consolidationService.validateConsolidationEligibility(primaryId, others); + if (validation.isValid) { + const preview = await consolidationService.previewConsolidation(primaryId, others); + console.log(preview.summary); -`DuplicateManager` also exposes helper methods: + const result = await consolidationService.consolidateMemories(primaryId, others); + console.log(`Merged ${result.consolidated} memories`); + } +} +``` -- `calculateContentSimilarity(a, b)` – returns a score between 0 and 1. Useful for manual checks. -- `detectDuplicateCandidates(namespace, { minSimilarity, limit })` – scans recent memories for likely duplicates without needing an explicit query. -- `validateConsolidationSafety(primaryId, duplicateIds)` – ensures memories exist and are in an appropriate state for consolidation. -- `getConsolidationHistory(namespace?)` – returns an in-memory log of previous operations performed during the current process lifetime. +The consolidation run is responsible for safety checks, atomic updates, and rollback support (see concrete methods on the `ConsolidationService` / `MemoryConsolidationService` implementations). -Import them directly when you need finer control: +## Supporting Utilities (Advanced/Internal) -```typescript -import { DuplicateManager } from 'memorits/core/infrastructure/database/DuplicateManager'; -``` +Internal components such as `DuplicateManager` and repository helpers expose lower-level utilities (e.g. candidate detection, consolidation validation, history/analytics). These are wired behind `Memori.findDuplicateMemories` and the consolidation service. -As with other internal imports, this path is subject to change until a public facade is provided. +If you choose to import them directly from internal paths (e.g. `src/core/infrastructure/database/DuplicateManager.ts`), treat this as unstable advanced usage: +- APIs and paths may change without notice. +- Prefer `Memori.findDuplicateMemories` and `memori.getConsolidationService()` for stable integration. ## Practical Workflow -1. Use `findDuplicateMemories` when ingesting new information to alert users about potential duplicates. -2. Schedule a periodic conscious consolidation run (`consolidateDuplicates`) if conscious mode is enabled. -3. Log or review similarity scores above a manual threshold before deleting or merging content. -4. Consider storing references in memory metadata (e.g., `metadata.relatedIds`) when a duplicate is intentionally retained. +1. Use `Memori.findDuplicateMemories` when ingesting new information to alert users about potential duplicates. +2. Use `memori.getConsolidationService()` for controlled consolidation flows (detect → validate → preview → consolidate → rollback if needed). +3. Log or review similarity scores and previews before deleting or merging content. +4. If you must call internal managers directly, isolate that logic in your own adapter and treat it as non-stable API surface. Duplicate detection is designed to be conservative: it surfaces likely matches without deleting anything automatically. This keeps the system safe by default while giving you the hooks to build richer moderation or review workflows. diff --git a/docs/developer/api/core-api.md b/docs/developer/api/core-api.md index affff06..e22d1ea 100644 --- a/docs/developer/api/core-api.md +++ b/docs/developer/api/core-api.md @@ -1,137 +1,430 @@ # Core API Reference -This reference covers the primary classes exported by Memorits: `MemoriAI`, `Memori`, and the OpenAI drop-in client. All types referenced below come directly from the TypeScript definitions in `src/`. +This guide shows how to use the main Memorits APIs from your application code: -## `MemoriAI` +- `Memori` – advanced control over ingestion, search, maintenance. +- `MemoriAI` – simplified facade for most app integrations. +- Configuration inputs – how to pass config and environment safely. -High-level façade that combines provider access and memory management. +All behavior described is backed by the implementation in [`src/core/Memori.ts`](src/core/Memori.ts:21), [`src/core/MemoriAI.ts`](src/core/MemoriAI.ts:37), and [`src/core/infrastructure/config/ConfigManager.ts`](src/core/infrastructure/config/ConfigManager.ts:29). -### Constructor +## Memori -```typescript -new MemoriAI(config: MemoriAIConfig) +Advanced API for memory orchestration, ingestion control, search, statistics, and maintenance. + +Location: +- [`src/core/Memori.ts`](src/core/Memori.ts:21) + +### Construction (how to create it in your app) + +```ts +new Memori(config?: Partial) ``` -`MemoriAIConfig` lives in `src/core/MemoriAIConfig.ts` and includes: +Key config behavior: +- Loads baseline from environment via `ConfigManager.loadConfig()`: + - [`ConfigManager`](src/core/infrastructure/config/ConfigManager.ts:29) +- If `config.mode` is provided: + - `automatic` → `autoIngest = true`, `consciousIngest = false` + - `manual` → both disabled + - `conscious` → `consciousIngest = true`, `autoIngest = false` +- Other fields (`databaseUrl`, `apiKey`, `baseUrl`, `namespace`, `userContext`, etc.) override defaults. -- `databaseUrl: string` -- `apiKey: string` -- `provider?: 'openai' | 'anthropic' | 'ollama'` -- `model?: string` -- `baseUrl?: string` -- `mode?: 'automatic' | 'manual' | 'conscious'` -- `namespace?: string` -- Optional `userProvider` / `memoryProvider` overrides with provider-specific configuration. +Errors: +- Synchronous construction does not throw for DB connectivity; config validation errors can be thrown by `ConfigManager`. -### Methods +### enable() -| Method | Description | -| --- | --- | -| `chat(params: ChatParams): Promise` | Performs a chat completion using the configured provider. In automatic mode the exchange is recorded as memory. | -| `searchMemories(query: string, options?: SearchOptions): Promise` | Searches stored memories using the simplified search options from `MemoriAIConfig`. | -| `searchMemoriesWithStrategy(query: string, strategy: SearchStrategy, options?: SearchOptions): Promise` | Forces a specific strategy (e.g., `SearchStrategy.RECENT`). | -| `createEmbeddings(params: EmbeddingParams): Promise` | Creates embeddings using the active provider. | -| `recordConversation(userInput: string, aiOutput: string, options?): Promise` | Available in manual/conscious mode to persist conversations explicitly. | -| `getMemoryStatistics(namespace?: string)` | Returns counts from `DatabaseManager.getMemoryStatistics`. | -| `getAvailableSearchStrategies(): Promise` | Lists the strategies currently registered by `SearchService`. | -| `getSessionId(): string` | Returns the generated session identifier for telemetry. | -| `getMode(): 'automatic' | 'manual' | 'conscious'` | Indicates the configured ingestion mode. | -| `close(): Promise` | Disposes provider resources and closes database connections. | +```ts +await memori.enable() +``` -### Types +Responsibilities: +- Initializes providers (dual-provider architecture). +- Creates: + - `DatabaseManager` + - `MemoryAgent` + - `ConsciousAgent` if `consciousIngest` is enabled. +- Starts background monitoring if conscious mode is active. -`ChatParams`, `ChatResponse`, `SearchOptions`, `MemorySearchResult`, `EmbeddingParams`, and `EmbeddingResponse` are exported from `src/core/MemoriAIConfig.ts`. The `SearchOptions` shape here is the simplified version (namespace, limit, includeMetadata, minImportance, categories, sortBy, offset). +Key properties: +- Throws if called when already enabled. +- Conscious initialization failures are logged but do not cause `enable()` to reject. -## `Memori` +Reference: +- [`Memori.enable`](src/core/Memori.ts:174) -Low-level API with access to advanced search, consolidation, and maintenance APIs. Constructor accepts a partial `MemoriAIConfig`; any omitted values fall back to `ConfigManager`. +### recordConversation() -```typescript -const memori = new Memori({ databaseUrl: 'file:./memori.db', mode: 'conscious' }); -await memori.enable(); +```ts +await memori.recordConversation(userInput, aiOutput, options?) ``` -### Key Methods +Behavior: +- Requires `enable()` to have been called; otherwise throws. +- Persists chat via `DatabaseManager.storeChatHistory`. +- Ingestion by mode: + - autoIngest: triggers asynchronous `processMemory`. + - consciousIngest: only stores; background/conscious agent processes later. + - neither: store-only. + +Errors: +- DB write errors propagate. +- In auto mode, memory processing errors are logged; the call resolves once chat is stored. -| Method | Description | -| --- | --- | -| `enable(): Promise` | Initialises providers, database managers, and agents. Must be called before other operations. | -| `recordConversation(userInput, aiOutput, options?)` | Stores an exchange and processes it via `MemoryAgent`. | -| `searchMemories(query, options: SearchOptions)` | Uses the full search options defined in `src/core/types/models.ts` (including `temporalFilters`, `metadataFilters`, `filterExpression`, etc.). | -| `searchMemoriesWithStrategy(query, strategy, options)` | Executes a specific `SearchStrategy`. | -| `searchRecentMemories(limit?, includeMetadata?, temporalFilters?, strategy?)` | Convenience wrapper for recent searches. | -| `getAvailableSearchStrategies()` | Lists registered strategies. | -| `getMemoryStatistics(namespace?)` / `getDetailedMemoryStatistics(namespace?)` | Returns aggregated database stats. | -| `getIndexHealthReport()` | Proxy to `SearchIndexManager.getIndexHealthReport`. | -| `optimizeIndex(type?)` | Runs index optimisation (`MERGE`, `REBUILD`, `COMPACT`, `VACUUM`). | -| `createIndexBackup()` / `restoreIndexFromBackup(backupId)` | Manage search index backups. | -| `findDuplicateMemories(content, options?)` | Surfaces potential duplicates using `DuplicateManager`. | -| `getConsolidationService()` | Exposes the fully-fledged consolidation service for advanced workflows. | -| `initializeConsciousContext()` / `checkForConsciousContextUpdates()` | Conscious processing helpers. | -| `close()` | Disposes provider and database resources. | +Reference: +- [`Memori.recordConversation`](src/core/Memori.ts:226) -Any method that reads or writes data requires `enable()` to have run first. +### processMemory() (internal-facing) -## `MemoriOpenAI` +```ts +// Called internally; exposed as method for advanced usage +await memori['processMemory'](chatId, userInput, aiOutput) +``` -Located under `integrations/openai-dropin/client`. Provides a drop-in replacement for the OpenAI SDK v5 while recording memories automatically. +Behavior: +- Ensures `MemoryAgent` is initialized. +- Calls `MemoryAgent.processConversation`. +- Persists `ProcessedLongTermMemory`. +- If `enableRelationshipExtraction` and relationships exist: + - Stores via `storeMemoryRelationships`. -```typescript -import { MemoriOpenAI } from 'memorits/integrations/openai-dropin/client'; +Guarantees: +- Logs and swallows errors (does not throw to caller of recordConversation in auto mode). -const client = new MemoriOpenAI({ - apiKey: process.env.OPENAI_API_KEY!, - model: 'gpt-4o-mini', - memory: { - enableChatMemory: true, - memoryProcessingMode: 'auto', - sessionId: 'support-bot' - } -}); +Reference: +- [`Memori.processMemory`](src/core/Memori.ts:277) -const completion = await client.chat.completions.create({ - model: 'gpt-4o-mini', - messages: [{ role: 'user', content: 'Remember that our stand-up is 9am PST.' }] -}); +### storeProcessedMemory() -const memories = await client.memory.searchMemories('stand-up'); +```ts +await memori.storeProcessedMemory(processedMemory, chatId, namespace?) ``` -The factory helpers (`MemoriOpenAIFactory`, `memoriOpenAIFactory`, `MemoriOpenAIFromConfig`, etc.) mirror the constructors defined in `integrations/openai-dropin/factory.ts`. +Use: +- For advanced flows where you run your own processing but still want to reuse Memori’s persistence and logging. + +Behavior: +- Requires `enable()` (Memori must be enabled). +- Ensures a `ChatHistory` row exists for the provided `chatId`, creating a minimal one if necessary. +- Persists long-term memory and logs metadata. -## Provider Utilities +Reference: +- [`Memori.storeProcessedMemory`](src/core/Memori.ts:655) -Provider abstractions live in `src/core/infrastructure/providers`. To initialise providers directly: +### Search APIs -```typescript -import { LLMProviderFactory, ProviderType } from 'memorits'; +#### searchMemories() -const provider = await LLMProviderFactory.createProvider(ProviderType.ANTHROPIC, { - apiKey: process.env.ANTHROPIC_API_KEY!, - model: 'claude-3-5-sonnet-20241022', - features: { - memory: { - enableChatMemory: true, - memoryProcessingMode: 'auto', - sessionId: 'anthropic-session' - } - } -}); +```ts +const results = await memori.searchMemories(query, options?) ``` -These utilities are considered advanced usage; most applications rely on `MemoriAI` or `Memori`. +Options (simplified): +- `limit` +- `minImportance` +- `categories` +- `includeMetadata` +- (internal SearchOptions may include temporal/metadata filters, strategy hints) + +Behavior: +- Delegates to `DatabaseManager.getSearchManager().searchMemories`. +- Uses configured strategies with FTS5 where available; falls back safely when not. + +Reference: +- [`Memori.searchMemories`](src/core/Memori.ts:371) + +#### searchMemoriesWithStrategy() + +```ts +const results = await memori.searchMemoriesWithStrategy(query, strategy, options?) +``` + +Behavior: +- Requires `enable()`. +- Uses `SearchService.searchWithStrategy`. +- Maps results into `MemorySearchResult` shape. + +Errors: +- Logs and throws a clear error if the strategy call fails. + +Reference: +- [`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:385) + +#### getAvailableSearchStrategies() + +```ts +const strategies = await memori.getAvailableSearchStrategies() +``` + +Reference: +- [`Memori.getAvailableSearchStrategies`](src/core/Memori.ts:435) + +#### searchRecentMemories() + +```ts +const results = await memori.searchRecentMemories(limit?, includeMetadata?, temporalOptions?, strategy?) +``` + +Behavior: +- Uses RECENT strategy or delegates to `searchMemoriesWithStrategy` when specified. + +Reference: +- [`Memori.searchRecentMemories`](src/core/Memori.ts:443) + +### Statistics + +#### getMemoryStatistics() + +```ts +const stats = await memori.getMemoryStatistics(namespace?) +``` + +Returns aggregate counts from `StatisticsManager`. + +Reference: +- [`Memori.getMemoryStatistics`](src/core/Memori.ts:886) + +#### getDetailedMemoryStatistics() + +```ts +const detailed = await memori.getDetailedMemoryStatistics(namespace?) +``` + +Includes breakdowns by type, importance, category, recent activity, and average confidence. + +Reference: +- [`Memori.getDetailedMemoryStatistics`](src/core/Memori.ts:927) + +### Duplicate & Relationship APIs + +#### findDuplicateMemories() + +```ts +const duplicates = await memori.findDuplicateMemories(content, { similarityThreshold?, namespace?, limit? }) +``` + +Behavior: +- Requires `enable()`. +- Delegates to `DuplicateManager` via `DatabaseManager`. +- Returns potential duplicates above threshold. + +Reference: +- [`Memori.findDuplicateMemories`](src/core/Memori.ts:735) + +#### extractMemoryRelationships() + +```ts +const rels = await memori.extractMemoryRelationships(content, options?) +``` + +- Advanced API; relies on `MemoryAgent` relationshipProcessor. +- Throws if MemoryAgent/RelationshipProcessor not available. + +Reference: +- [`Memori.extractMemoryRelationships`](src/core/Memori.ts:983) + +#### buildRelationshipGraph() + +```ts +const graph = await memori.buildRelationshipGraph(namespace?, { maxDepth?, includeWeakRelationships? }) +``` + +- Builds derived relationship graph using RelationshipProcessor via MemoryAgent. + +Reference: +- [`Memori.buildRelationshipGraph`](src/core/Memori.ts:1050) + +### Index Maintenance + +#### getIndexHealthReport(), optimizeIndex(), createIndexBackup(), restoreIndexFromBackup() + +- All require `enable()`. +- Delegate to `SearchIndexManager` via `DatabaseManager`. + +References: +- [`Memori.getIndexHealthReport`](src/core/Memori.ts:794) +- [`Memori.optimizeIndex`](src/core/Memori.ts:824) +- [`Memori.createIndexBackup`](src/core/Memori.ts:856) +- [`Memori.restoreIndexFromBackup`](src/core/Memori.ts:1112) + +### Lifecycle + +#### close() + +```ts +await memori.close() +``` + +Behavior: +- Stops background monitoring. +- Attempts to clean up SearchService. +- Closes DatabaseManager. + +Reference: +- [`Memori.close`](src/core/Memori.ts:467) + +--- + +## MemoriAI + +High-level, user-facing API that wraps a provider plus an internal Memori instance. + +Location: +- [`src/core/MemoriAI.ts`](src/core/MemoriAI.ts:37) + +### Construction + +```ts +new MemoriAI(config: MemoriAIConfig) +``` + +Key points: +- Generates `sessionId`. +- Detects provider via: + - `config.provider` (if set), or + - API key patterns / `ollama-local`. + - See [`MemoriAI.detectProvider`](src/core/MemoriAI.ts:313). +- Creates: + - User-facing provider for chat/embeddings. + - Internal Memori with mapped config (databaseUrl, apiKey, model, namespace, mode). + +### chat() + +```ts +const res = await memoriAI.chat(params: ChatParams) +``` + +Behavior: +- Sends chat to `userProvider.createChatCompletion`. +- In `mode: 'automatic'`: + - Ensures `Memori` is enabled. + - Calls `Memori.recordConversation` to persist dialogue. +- Returns normalized `ChatResponse`. + +Errors: +- Logs via `logError` with component `MemoriAI`. +- Propagates provider errors. + +Reference: +- [`MemoriAI.chat`](src/core/MemoriAI.ts:97) + +### searchMemories() + +```ts +const results = await memoriAI.searchMemories(query, options?) +``` + +- Converts options to internal shape. +- Delegates to `Memori.searchMemories`. + +Reference: +- [`MemoriAI.searchMemories`](src/core/MemoriAI.ts:153) + +### searchMemoriesWithStrategy(), getAvailableSearchStrategies() + +- Thin wrappers around Memori’s implementations. + +References: +- [`MemoriAI.searchMemoriesWithStrategy`](src/core/MemoriAI.ts:293) +- [`MemoriAI.getAvailableSearchStrategies`](src/core/MemoriAI.ts:306) + +### createEmbeddings() + +```ts +const res = await memoriAI.createEmbeddings(params: EmbeddingParams) +``` + +- Uses `userProvider.createEmbedding`. +- Returns normalized EmbeddingResponse. + +Reference: +- [`MemoriAI.createEmbeddings`](src/core/MemoriAI.ts:167) + +### recordConversation() (manual/conscious modes) + +```ts +const chatId = await memoriAI.recordConversation(userInput, aiOutput, options?) +``` + +- Only allowed when mode is `manual` or `conscious`. +- Forwards to `Memori.recordConversation`. + +Reference: +- [`MemoriAI.recordConversation`](src/core/MemoriAI.ts:247) + +### getMemoryStatistics() + +```ts +const stats = await memoriAI.getMemoryStatistics(namespace?) +``` + +- Delegates to `Memori.getMemoryStatistics`. + +Reference: +- [`MemoriAI.getMemoryStatistics`](src/core/MemoriAI.ts:286) + +### close() + +```ts +await memoriAI.close() +``` + +- Disposes user provider (if any). +- Closes underlying Memori. + +Reference: +- [`MemoriAI.close`](src/core/MemoriAI.ts:203) + +--- + +## Configuration Overview + +Memori and MemoriAI rely on two main configuration layers: + +1. MemoriAIConfig (user-facing) +2. MemoriConfig via ConfigManager (env-driven defaults) + +### MemoriAIConfig (selected fields) + +Defined in: +- [`src/core/MemoriAIConfig.ts`](src/core/MemoriAIConfig.ts) + +Key fields: +- `databaseUrl` +- `apiKey` +- `baseUrl` +- `model` +- `namespace` +- `mode`: 'automatic' | 'manual' | 'conscious' +- `provider` (optional hint) +- `memoryProvider` (optional advanced configuration for memory side) + +Used by: +- [`MemoriAI`](src/core/MemoriAI.ts:37) +- `Memori` (when constructed via MemoriAI config subset) -## Logging +### MemoriConfig via ConfigManager -All core classes use `logInfo`/`logError` from `src/core/infrastructure/config/Logger.ts`. Each log entry includes a `component` field (e.g., `MemoriAI`, `SearchManager`) and contextual metadata like `sessionId`, `namespace`, or operation identifiers. +Defined/loaded in: +- [`MemoriConfigSchema`](src/core/infrastructure/config/ConfigManager.ts:11) +- [`ConfigManager.loadConfig`](src/core/infrastructure/config/ConfigManager.ts:29) -## Error Handling +Environment variables: +- `DATABASE_URL` / `MEMORI_DATABASE_URL` +- `MEMORI_NAMESPACE` +- `MEMORI_CONSCIOUS_INGEST` +- `MEMORI_AUTO_INGEST` +- `MEMORI_ENABLE_RELATIONSHIP_EXTRACTION` +- `MEMORI_MODEL` +- `OPENAI_API_KEY` +- `OPENAI_BASE_URL` (incl. Ollama-style endpoints) -- Configuration errors throw `ValidationError` or `SanitizationError`. -- Provider initialisation errors bubble up from the underlying SDKs. -- Search operations throw when options fail validation or when fallbacks also fail. -- Consolidation and duplicate operations return structured results containing `errors`/`warnings`. +Rules: +- If no valid API key and no `OPENAI_BASE_URL`, throws configuration error. +- If `OPENAI_BASE_URL` is set without key, assigns `ollama-local` as synthetic key. +- All values sanitized and validated; throws `SanitizationError` / `ValidationError` on invalid input. -Always wrap calls in `try/catch` when building production systems to handle these outcomes gracefully. +--- -This reference should give you a working mental model of the public API. For deeper dives, consult the module-specific docs (`advanced-features/`, `core-concepts/`) or inspect the source files referenced above. +This core API reference is intended to be stable and code-accurate. Any symbol not listed here or in top-level exports should be treated as internal and subject to change. diff --git a/docs/developer/api/search-api.md b/docs/developer/api/search-api.md index 45df2b5..4d0a101 100644 --- a/docs/developer/api/search-api.md +++ b/docs/developer/api/search-api.md @@ -1,163 +1,322 @@ # Search API Reference -This reference documents the advanced search API backed by `Memori` and `SearchService`. It describes the `SearchOptions` type, available strategies, and sample queries grounded in the implementation under `src/core/domain/search`. +Authoritative reference for search-related inputs and behavior in Memorits. -## Search Options (`src/core/types/models.ts`) +This document describes: +- SearchOptions and related shapes +- Search strategy usage via Memori / MemoriAI +- Temporal and metadata filtering behavior (grounded in existing implementations) -```typescript -interface SearchOptions { - namespace?: string; - limit?: number; - includeMetadata?: boolean; +For core entrypoints, see: +- [`Memori.searchMemories`](src/core/Memori.ts:371) +- [`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:385) +- [`Memori.searchRecentMemories`](src/core/Memori.ts:443) +- [`MemoriAI.searchMemories`](src/core/MemoriAI.ts:153) +- [`MemoriAI.searchMemoriesWithStrategy`](src/core/MemoriAI.ts:293) - minImportance?: MemoryImportanceLevel; - categories?: MemoryClassification[]; - temporalFilters?: TemporalFilterOptions; - metadataFilters?: MetadataFilterOptions; +Only documented options and strategies here should be treated as stable. - sortBy?: { field: string; direction: 'asc' | 'desc' }; - offset?: number; +--- - strategy?: SearchStrategy; - timeout?: number; - enableCache?: boolean; +## Search Entry Points - filterExpression?: string; - includeRelatedMemories?: boolean; - maxRelationshipDepth?: number; -} +### Memori.searchMemories + +```ts +const results = await memori.searchMemories(query: string, options?: SearchOptions) ``` -Pass this structure to `Memori.searchMemories` or `Memori.searchMemoriesWithStrategy`. +- Delegates to `SearchManager.searchMemories`. +- Applies `namespace`, `limit`, `minImportance`, category filters, and optional filters supported by the underlying strategies. +- Used for: + - Standard keyword/semantic-like retrieval. + - Recent queries when combined with appropriate strategy hints. -## Strategies (`SearchStrategy` enum) +Reference: +- [`Memori.searchMemories`](src/core/Memori.ts:371) -```typescript -enum SearchStrategy { - FTS5 = 'fts5', - LIKE = 'like', - RECENT = 'recent', - SEMANTIC = 'semantic', // placeholder for future vector search backends - CATEGORY_FILTER = 'category_filter', - TEMPORAL_FILTER = 'temporal_filter', - METADATA_FILTER = 'metadata_filter', - RELATIONSHIP = 'relationship' -} +### Memori.searchMemoriesWithStrategy + +```ts +const results = await memori.searchMemoriesWithStrategy( + query: string, + strategy: SearchStrategy, + options?: SearchOptions +) ``` -Use `Memori.searchMemoriesWithStrategy(query, SearchStrategy.FTS5, options)` to force a specific path; otherwise the search layer selects one automatically. +- Requires `memori.enable()`. +- Delegates to `SearchService.searchWithStrategy`. +- Wraps results into the `MemorySearchResult` shape, including: + - `id`, `content`, `summary` + - `classification`, `importance` + - `confidenceScore`, `metadata.searchStrategy`, `metadata.searchScore`, etc. when `includeMetadata` is true. + +On failure: +- Logs details with `component: 'Memori'`. +- Throws `Error` with strategy name and underlying message. + +Reference: +- [`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:385) + +### Memori.searchRecentMemories + +```ts +const results = await memori.searchRecentMemories( + limit?: number, + includeMetadata?: boolean, + temporalOptions?: TemporalFilterOptions, + strategy?: SearchStrategy +) +``` -## Examples +- Convenience wrapper over: + - `searchMemoriesWithStrategy` (if non-RECENT strategy provided) + - or `searchMemories` with RECENT semantics / temporal filters. +- Use when primarily interested in recency/time windows. -### Importance and Category Filters +Reference: +- [`Memori.searchRecentMemories`](src/core/Memori.ts:443) -```typescript -const important = await memori.searchMemories('roadmap', { - minImportance: 'high', - categories: ['essential', 'reference'], - includeMetadata: true, - limit: 20 -}); -``` +### MemoriAI.searchMemories / searchMemoriesWithStrategy -### Temporal Filters +`MemoriAI` exposes the same capabilities via: -```typescript -const recentUpdates = await memori.searchMemories('launch', { - temporalFilters: { - relativeExpressions: ['last 14 days'] - }, - limit: 15 -}); +```ts +const results = await memoriAI.searchMemories(query, options) +const results = await memoriAI.searchMemoriesWithStrategy(query, strategy, options) ``` -`TemporalFilterOptions` supports `timeRanges`, `relativeExpressions`, `absoluteDates`, and `patterns`. Combine them as needed. +These: +- Normalize user-facing options. +- Delegate to the underlying `Memori` instance. -### Metadata Filters +References: +- [`MemoriAI.searchMemories`](src/core/MemoriAI.ts:153) +- [`MemoriAI.searchMemoriesWithStrategy`](src/core/MemoriAI.ts:293) -```typescript -const billingNotes = await memori.searchMemories('', { - metadataFilters: { - fields: [ - { key: 'metadata.topic', operator: 'eq', value: 'billing' }, - { key: 'metadata.importanceScore', operator: 'gte', value: 0.7 } - ] - }, - includeMetadata: true, - limit: 10 -}); -``` +--- -Supported operators: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`, `like`. +## SearchOptions (Conceptual) -### Filter Expressions +The concrete types live in: +- [`src/core/types/models.ts`](src/core/types/models.ts) +- [`src/core/domain/search/types`](src/core/domain/search/types.ts) -```typescript -const filtered = await memori.searchMemories('', { - filterExpression: 'importance_score >= 0.7 AND category = "essential"', - limit: 25 -}); +At a high level, `SearchOptions` supports: + +- `limit?: number` + - Max results to return (sensible defaults applied). +- `offset?: number` + - Optional pagination offset (if supported by strategy). +- `namespace?: string` + - Logical partition for multi-tenant setups. +- `minImportance?: string` + - Filter by minimum importance level. Mapped internally to `MemoryImportanceLevel`. +- `categories?: string[]` + - Filter by categories / classification (strategy-specific). +- `includeMetadata?: boolean` + - When true, includes strategy and scoring metadata in results. +- `temporalFilters?: TemporalFilterOptions` + - Enable time-aware filtering (see below). +- `metadataFilters?: MetadataFilterOptions` + - Enable structured metadata filtering (see below). +- Strategy hints (where supported): + - Strategy may interpret additional hints; only rely on those documented here. + +Any option not documented here or in the public types should be considered internal. + +--- + +## SearchResult Shape (MemorySearchResult) + +`MemorySearchResult` (simplified): + +- `id: string` +- `content: string` +- `summary?: string` +- `classification: MemoryClassification` +- `importance: MemoryImportanceLevel` +- `topic?: string` +- `confidenceScore?: number` +- `classificationReason?: string` +- `metadata?: { ... }` when `includeMetadata` is enabled: + - `searchScore?: number` + - `searchStrategy?: string` + - `memoryType?: string` + - `category?: string` + - `importanceScore?: number` + - `timeRange?` / other strategy-specific annotations when provided. + +Memori’s `searchMemoriesWithStrategy` uses these metadata fields explicitly: +- [`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:401) + +--- + +## Search Strategies + +Search strategies are defined in: +- [`src/core/domain/search/types`](src/core/domain/search/types.ts) +- Implementation spread across SearchManager, SearchService, and strategy classes. + +Common patterns: + +- `RECENT` + - Orders by recency; used by `searchRecentMemories`. +- Temporal / metadata / relationship strategies + - Implemented by dedicated strategy classes and wired via `SearchService`. + +This version of the docs intentionally: +- Treats the concrete list of strategies as coming from `getAvailableSearchStrategies()`: + - Use `await memori.getAvailableSearchStrategies()` or MemoriAI equivalent to discover supported strategies at runtime. +- Does not hardcode strategy names beyond RECENT/common ones to avoid divergence. + +Reference: +- [`Memori.getAvailableSearchStrategies`](src/core/Memori.ts:435) + +When implementing: +- Prefer using `getAvailableSearchStrategies()` to branch on capabilities. +- Handle unknown strategies defensively. + +--- + +## Temporal Filtering + +Temporal filtering is configured via `temporalFilters` in `SearchOptions`. + +Type reference: +- See `TemporalFilterOptions` in [`src/core/types/models.ts`](src/core/types/models.ts) +- Detailed usage: [`docs/developer/advanced-features/temporal-filtering.md`](docs/developer/advanced-features/temporal-filtering.md) + +Conceptual shape: + +```ts +interface TemporalFilterOptions { + timeRanges?: Array<{ start: Date; end: Date }>; + relativeExpressions?: string[]; + absoluteDates?: Date[]; + patterns?: string[]; +} ``` -`filterExpression` uses the `AdvancedFilterEngine` parser and accepts SQL-like syntax with `AND`, `OR`, `NOT`, comparison operators, and quoted string literals. +Behavior: +- Strategies interpret these options using: + - `DateTimeNormalizer` and related helpers. +- Typical use: + - Restrict results to: + - Last N hours/days/weeks. + - Specific calendar ranges. +- When combined with RECENT-like strategies, temporal filters constrain candidate sets. -### Relationship Search +Usage example (Memori): -```typescript -const related = await memori.searchMemories('incident response', { - includeRelatedMemories: true, - maxRelationshipDepth: 2, - includeMetadata: true +```ts +const recent = await memori.searchMemories('deployment', { + temporalFilters: { relativeExpressions: ['last 48 hours'] }, + includeMetadata: true, + limit: 20, }); ``` -When `includeRelatedMemories` is true the relationship strategy augments results with linked memories captured by `MemoryAgent`. +Guarantees: +- If a temporal strategy is active, results are biased/filtered by time. +- If unsupported, temporal filters are ignored or handled conservatively; inspect `metadata` when `includeMetadata` to confirm. -### Recent Helper +--- -```typescript -const recent = await memori.searchRecentMemories( - 10, - true, - { relativeExpressions: ['today'] } -); -``` +## Metadata Filtering -This convenience method maps to the temporal strategy internally. - -## Result Structure (`MemorySearchResult`) - -```typescript -interface MemorySearchResult { - id: string; - content: string; - summary: string; - classification: MemoryClassification; - importance: MemoryImportanceLevel; - topic?: string; - entities: string[]; - keywords: string[]; - confidenceScore: number; - classificationReason: string; - metadata?: Record; -} -``` +Metadata filtering is configured via `metadataFilters` in `SearchOptions`. -- `metadata.searchStrategy` identifies the strategy used when `includeMetadata` is true. -- `metadata.searchScore` contains the raw score emitted by the strategy (when available). -- `confidenceScore` is the value stored with the memory, not necessarily the search similarity. +Type reference: +- See `MetadataFilterOptions` in [`src/core/types/models.ts`](src/core/types/models.ts) +- Detailed usage: [`docs/developer/advanced-features/metadata-filtering.md`](docs/developer/advanced-features/metadata-filtering.md) -## Error Handling +Conceptual shape: -- Invalid search options trigger `ValidationError` from `SearchManager`. -- When a primary strategy fails, a fallback strategy runs automatically; errors are logged via `SearchManager`. -- Explicit strategy calls (`searchMemoriesWithStrategy`) throw if the strategy is unknown or unavailable. +```ts +interface MetadataFilterOptions { + fields?: Array<{ + key: string; + value: unknown; + operator: 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte' | 'in' | 'contains' | 'like'; + }>; +} +``` + +Behavior: +- Strategy builds safe SQL/JSON expressions based on: + - Key (supports dot-notation for nested metadata). + - Operator and value. +- Supports combining multiple fields (AND semantics in current implementation). +- Can be combined with: + - Text query + - Temporal filters + - Other strategy logic -## Performance Tips +Usage example: -- Use namespaces to segment tenants and avoid scanning unrelated data. -- Set `limit` to a reasonable number—strategies cap results at 100 by default. -- Include metadata only when needed to reduce payload size. -- Combine `filterExpression` with `temporalFilters` to narrow result sets before they reach the relationship strategy. +```ts +const filtered = await memori.searchMemories('', { + metadataFilters: { + fields: [ + { key: 'metadata.importanceScore', operator: 'gte', value: 0.7 }, + { key: 'metadata.category', operator: 'in', value: ['operations', 'security'] }, + ], + }, + includeMetadata: true, + limit: 10, +}); +``` -For more context on how strategies work together, see `core-concepts/search-strategies.md` and `architecture/search-architecture.md`. +Guarantees: +- All keys/values are sanitized. +- Invalid operators/keys are rejected or ignored according to implementation; treat this API as strict. + +--- + +## Combining Filters and Strategies + +Recommended patterns: + +- For basic keyword search: + - Use `searchMemories(query, { limit, minImportance })`. +- For recent-focused: + - Use `searchRecentMemories` or a RECENT-like strategy via `searchMemoriesWithStrategy`. +- For time + structure: + - Combine `temporalFilters` + `metadataFilters` with an appropriate strategy. +- For advanced scenarios: + - Discover strategies via `getAvailableSearchStrategies()`. + - Route queries accordingly. + +Implementation notes: +- FTS5 vs LIKE: + - When FTS5 is available and initialized, strategies may use it for scoring. + - When not, LIKE/JSON-based fallbacks are used. + - Callers should not depend on a specific backend, only on the documented behavior: + - Relevant matches within the given constraints, + - Stable result shape. + +--- + +## Stability Notes + +- Public and stable: + - `Memori.searchMemories` + - `Memori.searchMemoriesWithStrategy` + - `Memori.searchRecentMemories` + - `Memori.getAvailableSearchStrategies` + - `MemoriAI.searchMemories` + - `MemoriAI.searchMemoriesWithStrategy` + - `SearchOptions` fields documented above + - `TemporalFilterOptions` and `MetadataFilterOptions` as documented + +- Advanced but supported: + - Strategy-specific metadata in `MemorySearchResult.metadata`. + - Using `getAvailableSearchStrategies()` to branch on capabilities. + +- Internal: + - Direct imports from `src/core/domain/search/**`, `SearchManager`, and individual strategies. + - Callers should not rely on internal class names or their exact signatures; use the high-level APIs above instead. + +This reference is intended to stay synchronized with the implementation; any discrepancy should be treated as a bug in the documentation. diff --git a/docs/developer/architecture/system-overview.md b/docs/developer/architecture/system-overview.md index 3258ac6..e6db907 100644 --- a/docs/developer/architecture/system-overview.md +++ b/docs/developer/architecture/system-overview.md @@ -43,31 +43,65 @@ Each stage logs structured payloads using `Logger.ts`, tagging the component (e. ### `MemoriAI` - Location: `src/core/MemoriAI.ts` -- Designed for consumers who want a drop-in API: `chat`, `searchMemories`, `createEmbeddings`, `recordConversation` (manual mode), `getMemoryStatistics`, etc. -- Delegates most heavy lifting to `Memori` while managing provider lifecycles. +- Public surface (as implemented): + - `chat` – chat completion with optional automatic recording via `Memori` ([`MemoriAI.chat`](src/core/MemoriAI.ts:97)) + - `searchMemories` / `searchMemoriesWithStrategy` – delegates to `Memori` search APIs ([`MemoriAI.searchMemories`](src/core/MemoriAI.ts:153), [`MemoriAI.searchMemoriesWithStrategy`](src/core/MemoriAI.ts:293)) + - `getAvailableSearchStrategies` – enumerates supported strategies ([`MemoriAI.getAvailableSearchStrategies`](src/core/MemoriAI.ts:306)) + - `createEmbeddings` – provider-backed embeddings API ([`MemoriAI.createEmbeddings`](src/core/MemoriAI.ts:167)) + - `recordConversation` – manual/conscious mode recording wrapper ([`MemoriAI.recordConversation`](src/core/MemoriAI.ts:247)) + - `getMemoryStatistics` – statistics via underlying `Memori` ([`MemoriAI.getMemoryStatistics`](src/core/MemoriAI.ts:286)) + - `close` – disposes provider and `Memori` ([`MemoriAI.close`](src/core/MemoriAI.ts:203)) +- Delegates heavy lifting (ingestion, search, consolidation) to `Memori` while owning user-facing provider lifecycle. ### `Memori` - Location: `src/core/Memori.ts` -- Exposes the full surface: strategy-aware search, conscious processing, index maintenance, duplication checks, and backup/restore helpers. +- Key responsibilities (all verified in code): + - Ingestion: + - `enable` – initializes providers, `MemoryAgent`, optional `ConsciousAgent` ([`Memori.enable`](src/core/Memori.ts:174)) + - `recordConversation` – stores chat + triggers ingestion based on mode ([`Memori.recordConversation`](src/core/Memori.ts:226)) + - `processMemory` – runs `MemoryAgent.processConversation` + stores relationships when enabled ([`Memori.processMemory`](src/core/Memori.ts:277)) + - `storeProcessedMemory` – helper to persist externally computed memories ([`Memori.storeProcessedMemory`](src/core/Memori.ts:655)) + - Search: + - `searchMemories` ([`Memori.searchMemories`](src/core/Memori.ts:371)) + - `searchMemoriesWithStrategy` ([`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:385)) + - `searchRecentMemories` ([`Memori.searchRecentMemories`](src/core/Memori.ts:443)) + - `getAvailableSearchStrategies` ([`Memori.getAvailableSearchStrategies`](src/core/Memori.ts:435)) + - Conscious processing: + - `initializeConsciousContext`, `checkForConsciousContextUpdates` ([`Memori.initializeConsciousContext`](src/core/Memori.ts:534), [`Memori.checkForConsciousContextUpdates`](src/core/Memori.ts:514)) + - Background monitoring controls: `setBackgroundUpdateInterval`, `isBackgroundMonitoringActive` ([`Memori.setBackgroundUpdateInterval`](src/core/Memori.ts:617), [`Memori.isBackgroundMonitoringActive`](src/core/Memori.ts:641)) + - Index maintenance: + - `getIndexHealthReport`, `optimizeIndex`, `createIndexBackup`, `restoreIndexFromBackup` ([`Memori`](src/core/Memori.ts:794), [`Memori.optimizeIndex`](src/core/Memori.ts:824), [`Memori.createIndexBackup`](src/core/Memori.ts:856), [`Memori.restoreIndexFromBackup`](src/core/Memori.ts:1112)) + - Stats: + - `getMemoryStatistics`, `getDetailedMemoryStatistics` ([`Memori.getMemoryStatistics`](src/core/Memori.ts:886), [`Memori.getDetailedMemoryStatistics`](src/core/Memori.ts:927)) + - Duplicates & relationships: + - `findDuplicateMemories` ([`Memori.findDuplicateMemories`](src/core/Memori.ts:735)) + - `extractMemoryRelationships`, `buildRelationshipGraph` ([`Memori.extractMemoryRelationships`](src/core/Memori.ts:983), [`Memori.buildRelationshipGraph`](src/core/Memori.ts:1050)) + - Lifecycle: + - `close` – shuts down background tasks, search service, and database ([`Memori.close`](src/core/Memori.ts:467)) - Instantiates `DatabaseManager`, provider adapters, `MemoryAgent`, and optionally `ConsciousAgent`. ### Providers - Location: `src/core/infrastructure/providers/` -- `OpenAIProvider`, `AnthropicProvider`, `OllamaProvider` all extend `MemoryCapableProvider`. -- `LLMProviderFactory` converts `ProviderType` + config into the correct provider instance. -- Providers share connection pooling, request caching, and health monitoring utilities. +- `OpenAIProvider`, `AnthropicProvider`, `OllamaProvider` extend shared base provider infrastructure (e.g. `MemoryCapableProvider` / `BaseLLMProvider`). +- `LLMProviderFactory` maps `ProviderType` + config to concrete provider classes. +- Documented guarantees: + - Consistent config surface and logging. + - Memory features integrated via `MemoryCapableProvider` when enabled. +- Do not assume undocumented features like global caching or pooling beyond what is implemented in each provider. ### Database Infrastructure - `DatabaseContext` owns Prisma clients and operation metrics. -- `MemoryManager`, `SearchManager`, and supporting managers inherit from `BaseDatabaseService` to share sanitisation and metrics recording. -- Search coordination combines FTS5 queries with fallback strategies; see `SearchManager.searchMemories`. +- `MemoryManager`, `SearchManager`, and other managers are composed by `DatabaseManager` as a facade; refer to concrete types rather than assuming a specific base class. +- Search coordination combines FTS5 queries where available with safe fallbacks; see `SearchManager.searchMemories` and `SearchService` orchestration. ### Performance Monitoring -- Services under `src/core/performance/` (`PerformanceDashboardService`, `PerformanceAnalyticsService`) collect metrics from search, database operations, and configuration changes. They are optional but useful for admin dashboards. +- `PerformanceService` (wired through `DatabaseContext` / `DatabaseManager`) tracks database operation metrics and exposes: + - `getPerformanceMetrics`, `getRecentOperationMetrics`, `getPerformanceAnalytics`, `getDatabasePerformanceReport`, `getPerformanceMonitoringStatus`. +- These can be surfaced via `DatabaseManager`-backed APIs for admin tooling; avoid referencing non-existent `performance` dashboard classes. ### Integrations diff --git a/docs/developer/core-concepts/memory-management.md b/docs/developer/core-concepts/memory-management.md index e7405ef..06101b4 100644 --- a/docs/developer/core-concepts/memory-management.md +++ b/docs/developer/core-concepts/memory-management.md @@ -9,8 +9,8 @@ Memorits supports three ingestion styles. Each mode maps to concrete behaviour i ### Automatic (default) - Enabled when `mode: 'automatic'` or `MEMORI_AUTO_INGEST=true`. -- `MemoriAI.chat` calls `Memori.recordConversation` immediately after every provider response. -- `Memori.enable()` ensures the providers and database manager are initialised before recording. +- `MemoriAI.chat` uses its configured provider for inference and, in `mode: 'automatic'`, enables `Memori` (if needed) and calls `Memori.recordConversation` to persist the exchange ([`MemoriAI.chat`](src/core/MemoriAI.ts:97)). +- `Memori` maps `mode` → `autoIngest` / `consciousIngest` in its constructor and `enable()` ensures providers and `DatabaseManager` are initialised before ingestion ([`Memori`](src/core/Memori.ts:36), [`Memori.enable`](src/core/Memori.ts:174)). - Use for assistants that should remember everything without additional wiring. ### Manual @@ -48,7 +48,7 @@ Fields to note: - `isPermanentContext` – indicates the memory should persist across sessions. - References the originating conversation through `chatId`. -The `DatabaseManager.storeShortTermMemory` path is used by `ConsciousAgent` when promoting context. +Short-term memory promotion is handled via `ConsciousMemoryManager` and related helpers composed inside `DatabaseManager` (see conscious memory methods in [`DatabaseManager`](src/core/infrastructure/database/DatabaseManager.ts:741)). ### Long-Term Memory (`long_term_memory`) @@ -113,8 +113,8 @@ await memori.checkForConsciousContextUpdates(); ## Relationship Extraction and Duplication Control -- Relationship extraction is toggled by `enableRelationshipExtraction` (`true` by default in `ConfigManager`). -- Duplicate detection and consolidation live inside `Memori.findDuplicateMemories` and related helpers (see `src/core/Memori.ts` around the 700-line mark). Use them to prevent redundant long-term entries. +- Relationship extraction is toggled by `enableRelationshipExtraction` (default comes from `MEMORI_ENABLE_RELATIONSHIP_EXTRACTION`, see [`ConfigManager`](src/core/infrastructure/config/ConfigManager.ts:56)). +- Duplicate detection is exposed via `Memori.findDuplicateMemories` (delegating to `DuplicateManager` through `DatabaseManager`) ([`Memori.findDuplicateMemories`](src/core/Memori.ts:735)). Use this high-level API instead of relying on internal services. ```typescript const duplicates = await memori.findDuplicateMemories('mem_123', { similarityThreshold: 0.75 }); diff --git a/docs/developer/core-concepts/search-strategies.md b/docs/developer/core-concepts/search-strategies.md index 776b578..2a65e1a 100644 --- a/docs/developer/core-concepts/search-strategies.md +++ b/docs/developer/core-concepts/search-strategies.md @@ -1,240 +1,208 @@ -# Search Strategies in Memorits +# Search Strategies -The search stack is implemented under `src/core/domain/search`. `Memori` and `MemoriAI` both delegate to the `SearchManager` / `SearchService` combination and ultimately return `MemorySearchResult` objects. This document explains what each strategy does and how to select it deliberately. +This document explains how Memorits search strategies work and how to use them via stable public APIs. -## Strategy Catalogue +It is grounded in: +- [`Memori.searchMemories`](src/core/Memori.ts:371) +- [`Memori.searchMemoriesWithStrategy`](src/core/Memori.ts:385) +- [`Memori.searchRecentMemories`](src/core/Memori.ts:443) +- [`Memori.getAvailableSearchStrategies`](src/core/Memori.ts:435) +- [`MemoriAI.searchMemories`](src/core/MemoriAI.ts:153) +- [`MemoriAI.searchMemoriesWithStrategy`](src/core/MemoriAI.ts:293) +- Search infrastructure under `src/core/infrastructure/database` and `src/core/domain/search` -`SearchStrategy` is defined in `src/core/domain/search/types.ts`: +Only behavior that exists in the codebase is documented here. Anything else should be treated as internal. -```typescript -enum SearchStrategy { - FTS5 = 'fts5', - LIKE = 'like', - RECENT = 'recent', - SEMANTIC = 'semantic', - CATEGORY_FILTER = 'category_filter', - TEMPORAL_FILTER = 'temporal_filter', - METADATA_FILTER = 'metadata_filter', - RELATIONSHIP = 'relationship' - } - ``` - - ## Category and Hierarchy Support - - The search system provides sophisticated category management through the `CategoryMetadataExtractor` and `CategoryHierarchyManager` components. These work together to provide: - - ### Hierarchical Category Extraction - - The `CategoryMetadataExtractor` automatically detects categories from memory content and assigns them hierarchical paths: - - ```typescript - import { CategoryMetadataExtractor, CategoryHierarchyManager } from 'memorits'; - - // Create hierarchy manager with category structure - const hierarchyManager = new CategoryHierarchyManager(); - hierarchyManager.addCategory('Programming/Languages'); - hierarchyManager.addCategory('Programming/Frameworks'); - hierarchyManager.addCategory('Technology/Databases'); - - // Create extractor with hierarchy integration - const extractor = new CategoryMetadataExtractor(hierarchyManager); - - // Extract categories with hierarchy paths - const result = await extractor.extractCategories({ - content: 'I love working with Python and React for web development' - }); - - console.log(result.categories[0]); - // { - // name: 'Languages', - // hierarchyPath: 'Programming/Languages', // ✅ Hierarchical context - // confidence: 0.8, - // source: 'pattern', - // normalizedName: 'languages', - // relevanceScore: 0.9 - // } - ``` - - ### Category Hierarchy Management - - The `CategoryHierarchyManager` provides advanced operations for working with category hierarchies: - - ```typescript - // Build hierarchy from category paths - const categories = ['Programming/Languages', 'Programming/Frameworks', 'Technology/Cloud']; - const root = hierarchyManager.buildHierarchy(categories); - - // Traverse hierarchy - const descendants = hierarchyManager.getDescendants('Programming'); - // Returns: ['Languages', 'Frameworks'] and their children - - const ancestors = hierarchyManager.getAncestors('Languages'); - // Returns: [{ name: 'Programming', fullPath: 'Programming', ... }] - - // Search and validation - const matches = hierarchyManager.searchCategories('frame', 5); - const validation = hierarchyManager.validateHierarchy(); - ``` - - ### Advanced Category Features - - - **Pattern-Based Hierarchy**: Automatic hierarchy suggestion based on extraction patterns - - **Virtual Hierarchies**: Support for categories not in the main hierarchy - - **Batch Processing**: Efficient hierarchy operations for multiple categories - - **Related Categories**: Smart suggestions based on hierarchy relationships - - **Validation**: Consistency checking for hierarchy structures - - ### Category-Based Search - - Use category filters with hierarchy awareness: - - ```typescript - // Search with category filtering - const results = await memori.searchMemories('JavaScript tutorial', { - includeMetadata: true, - strategy: SearchStrategy.CATEGORY_FILTER - }); - - // Results include hierarchical category information - console.log(results[0].metadata.categories); - // [ - // { - // name: 'Languages', - // hierarchyPath: 'Programming/Languages', - // confidence: 0.8, - // relevanceScore: 0.9 - // } - // ] - ``` - -Each strategy lives in its own class (see `src/core/domain/search/strategies` and sibling folders) and is registered by `SearchService` when the necessary prerequisites are met (for example, the FTS5 strategy is skipped when SQLite lacks FTS5). - -## Default Behaviour +--- -`Memori.searchMemories(query, options)` chooses a strategy automatically: +## 1. Entry Points -- When FTS5 is available, keyword queries run through the FTS strategy. -- Empty queries or those dominated by time filters fall back to the recent or temporal strategies. -- The LIKE strategy is used as a fallback when FTS5 fails or is unavailable. +### 1.1 Using Memori -You will always receive an array of `MemorySearchResult` objects. When `includeMetadata` is `true`, the `metadata` field includes `searchScore`, `searchStrategy`, `memoryType`, and other diagnostic values. +- Default search: -## Selecting a Strategy Explicitly +```ts +const results = await memori.searchMemories('deployment notes', { + limit: 20, + minImportance: 'medium', + includeMetadata: true, +}); +``` -```typescript -import { Memori, SearchStrategy } from 'memorits'; +- Strategy-based search: -const memori = new Memori({ databaseUrl: 'file:./memori.db' }); -await memori.enable(); +```ts +import { SearchStrategy } from 'memorits'; // from public types if exported const results = await memori.searchMemoriesWithStrategy( - 'vector indexing', - SearchStrategy.FTS5, + 'incident report', + SearchStrategy.RECENT, { - limit: 20, - includeMetadata: true + limit: 50, + includeMetadata: true, } ); ``` -If a strategy throws, the `SearchManager` attempts a fallback, so you still receive results when possible. Check `metadata.searchStrategy` to see what executed in the end. +- Recent helper: -## Temporal Filtering +```ts +const recent = await memori.searchRecentMemories( + 10, + true, + { relativeExpressions: ['last 24 hours'] } +); +``` -`TemporalFilterOptions` live in `src/core/types/models.ts`: +### 1.2 Using MemoriAI -```typescript -const temporal = await memori.searchMemories('standup notes', { - limit: 10, - temporalFilters: { - relativeExpressions: ['last 7 days'], - absoluteDates: [new Date('2024-06-01')], - timeRanges: [ - { start: new Date('2024-05-01'), end: new Date('2024-05-15') } - ] - } -}); +MemoriAI forwards to Memori: + +```ts +const results = await memoriAI.searchMemories('project x', { limit: 10 }); +const withStrategy = await memoriAI.searchMemoriesWithStrategy('project x', strategy, { limit: 10 }); ``` -Supplying temporal filters automatically biases the strategy selection toward `TEMPORAL_FILTER` or `RECENT`. You can override this by passing `strategy: SearchStrategy.TEMPORAL_FILTER`. +Use MemoriAI when you want a single object for chat + memory + search. -## Metadata Filtering +--- -```typescript -const filtered = await memori.searchMemories('renewal', { - metadataFilters: { - fields: [ - { key: 'metadata.topic', operator: 'eq', value: 'billing' }, - { key: 'metadata.accountTier', operator: 'in', value: ['enterprise', 'pro'] } - ] - }, - includeMetadata: true -}); -``` +## 2. Discovering Supported Strategies -Metadata filters run through `MetadataFilterStrategy` and can be combined with the text query. When you need more expressive logic, use `filterExpression` which is parsed by `AdvancedFilterEngine`. +Instead of hardcoding strategy names, use: -```typescript -const advanced = await memori.searchMemories('', { - filterExpression: 'importanceScore >= 0.7 AND metadata.topic = "operations"', - limit: 25 -}); +```ts +const strategies = await memori.getAvailableSearchStrategies(); +// or +const strategies = await memoriAI.getAvailableSearchStrategies(); ``` -## Relationship Search +Treat this as the source of truth for what the current build supports. -The relationship strategy traverses the relationship graph generated during memory processing. +Reference: +- [`Memori.getAvailableSearchStrategies`](src/core/Memori.ts:435) -```typescript -const related = await memori.searchMemoriesWithStrategy( - 'incident response', - SearchStrategy.RELATIONSHIP, - { - limit: 15, - includeMetadata: true, - includeRelatedMemories: true, - maxRelationshipDepth: 2 - } -); -``` +Recommended pattern: +- Check `getAvailableSearchStrategies()` at startup. +- Enable/disable features in your app based on what’s available. -When `includeRelatedMemories` is `true`, additional related entries are appended to the result set. The underlying relationships come from `MemoryRelationship` objects stored in the database. +--- -## Recent Memory Helper +## 3. Common Strategy Behaviors -```typescript -const recent = await memori.searchRecentMemories( - 10, - true, - { - relativeExpressions: ['today'] - } -); -``` +The exact list of strategies is implementation-defined. However, several behaviors are stable and relied upon by higher-level APIs. -This helper wraps the temporal strategy. It is exposed on `Memori` and internally calls `searchMemoriesWithStrategy` on your behalf. +### 3.1 RECENT / Time-Aware Search -## Strategy Configuration +Usage: +- `Memori.searchRecentMemories` +- `searchMemoriesWithStrategy` with a RECENT-like strategy (when available). -`SearchStrategyConfigManager` (see `src/core/domain/search/SearchStrategyConfigManager.ts`) stores per-strategy configuration such as timeouts, scoring weights, and cache settings. Use `memori.getAvailableSearchStrategies()` to see what is active, and inspect the configuration manager if you need to adjust priorities or enable/disable strategies at runtime. +Behavior: +- Focuses on most recent memories in the target namespace. +- Can combine with `TemporalFilterOptions` to constrain time windows. +- Typical use cases: + - "What happened in the last 24h?" + - Dashboards and activity feeds. -## Error Handling and Fallbacks +See: +- [`docs/developer/advanced-features/temporal-filtering.md`](docs/developer/advanced-features/temporal-filtering.md) -Strategies instrument their execution with `logInfo`/`logError` calls. When a strategy fails: +### 3.2 Temporal Filtering (Strategy-Agnostic) -1. The failure is logged with the component name `SearchManager`. -2. `SearchManager` selects a fallback (usually LIKE) and re-runs the query. -3. Error statistics are recorded so you can inspect performance later via `getIndexHealthReport`. +Enabled via `temporalFilters` in `SearchOptions`: -Always wrap search calls in try/catch when building production systems: +- When a temporal-aware strategy is active: + - Filters / ranks using time intervals, relative expressions, etc. +- When not: + - Temporal options may be ignored or used conservatively. -```typescript -try { - const results = await memori.searchMemories('customer escalation'); - // ... -} catch (error) { - // Decide whether to surface the error or retry with relaxed filters -} -``` +Key points: +- Represents supported behavior; callers should inspect metadata when `includeMetadata` is enabled to confirm which strategy ran. + +Details: +- [`docs/developer/advanced-features/temporal-filtering.md`](docs/developer/advanced-features/temporal-filtering.md) + +### 3.3 Metadata Filtering + +Enabled via `metadataFilters` in `SearchOptions`: + +- Filters based on structured metadata stored with memories. +- Supports operators such as `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in`, `contains`, `like` according to the runtime implementation. + +Usage: +- Narrow results for operational cases: + - Importance thresholds. + - Specific categories. + - Tenant/account-level filters. + +Details: +- [`docs/developer/advanced-features/metadata-filtering.md`](docs/developer/advanced-features/metadata-filtering.md) + +--- + +## 4. FTS5 vs LIKE Fallbacks + +The search layer is designed to support environments: + +- With FTS5 enabled: + - Strategies may use FTS5-backed virtual tables for efficient full-text search and ranking. +- Without FTS5: + - Fallback to: + - Standard LIKE queries + - Other safe filters implemented in `SearchManager` / `SearchService`. + +Important: +- As a caller, you SHOULD NOT depend on a specific underlying mechanism. +- You SHOULD rely on these guarantees: + - Queries respect `namespace`, `limit`, documented filters. + - When FTS5 is unavailable, you still get functionally correct (if less sophisticated) results. +- To detect FTS status, use the database/index health APIs rather than assuming availability. + +--- + +## 5. Recommended Usage Patterns + +1. Basic keyword or general search: + - Use `searchMemories(query, { limit, minImportance })`. +2. Time-focused: + - Use `searchRecentMemories` or a RECENT strategy. + - Optionally pass `temporalFilters`. +3. Structured filters: + - Add `metadataFilters` and/or categories to narrow by metadata. +4. Adaptive strategy selection: + - On startup, call `getAvailableSearchStrategies()`. + - Route advanced queries only through known strategies. +5. Debugging behavior: + - Use `includeMetadata: true` to inspect: + - `searchStrategy` + - `searchScore` + - Any time or filter information surfaced in metadata. + +--- + +## 6. Stability Notes + +Stable and recommended: +- `Memori.searchMemories` +- `Memori.searchMemoriesWithStrategy` +- `Memori.searchRecentMemories` +- `Memori.getAvailableSearchStrategies` +- `MemoriAI.searchMemories` +- `MemoriAI.searchMemoriesWithStrategy` +- `SearchOptions` fields used in: + - [`docs/developer/api/search-api.md`](docs/developer/api/search-api.md) + +Advanced but supported: +- Inspecting `MemorySearchResult.metadata` when `includeMetadata` is true. +- Adapting behavior based on `getAvailableSearchStrategies()`. + +Internal / not guaranteed: +- Direct imports from: + - `src/core/domain/search/**` + - `SearchManager`, `SearchService` implementations + - Concrete strategy classes + +If you build on internal classes, treat them as unstable and wrap them in your own abstraction. -Armed with this knowledge you can mix and match strategies to suit your use case while staying aligned with the actual implementation inside the repository. +This document is intended to remain synchronized with the implementation; discrepancies should be corrected as documentation bugs. diff --git a/docs/developer/index.md b/docs/developer/index.md index d30d0b3..3cb8a2b 100644 --- a/docs/developer/index.md +++ b/docs/developer/index.md @@ -7,7 +7,7 @@ Welcome to the Memorits developer hub. Memorits is a TypeScript-first memory eng - **Agent-Friendly Memory** – `MemoriAI` captures conversations, scores importance, and stores summaries automatically. - **Prisma + SQLite Core** – A fully typed persistence layer with migrations managed through Prisma. - **Provider Abstraction** – Shared MemoryAgent pipelines for OpenAI, Anthropic, and Ollama via a unified provider factory. -- **Search Engine** – FTS5, recency, metadata, temporal, and relationship strategies with configurable fallbacks. +- **Search Engine** – Multi-strategy search via `SearchManager`/`SearchService`, using FTS5 where available with LIKE-based fallbacks, plus temporal and metadata filtering as implemented in the core search modules. - **Structured Logging** – All components log with component metadata, ready for production observability. - **Type-Safe APIs** – Zod-backed schemas drive runtime validation, and all public types are exported for IDE support. @@ -27,7 +27,10 @@ import { MemoriAI } from 'memorits'; const ai = new MemoriAI({ databaseUrl: 'file:./memori.db', apiKey: process.env.OPENAI_API_KEY ?? 'sk-your-api-key', - provider: 'openai', + // Provider is auto-detected: + // - sk-ant-* -> Anthropic + // - ollama-local/baseUrl -> Ollama + // - other sk-* -> OpenAI (default) model: 'gpt-4o-mini', mode: 'automatic', // automatic | manual | conscious namespace: 'support-bot' // optional logical partition @@ -40,7 +43,7 @@ const reply = await ai.chat({ const memories = await ai.searchMemories('TypeScript', { limit: 5 }); ``` -For Ollama or custom endpoints, supply `baseUrl` and the synthetic API key (`ollama-local`) that the providers expect. The `ConfigManager` also recognises environment settings such as `DATABASE_URL`, `MEMORI_NAMESPACE`, `MEMORI_AUTO_INGEST`, and `OPENAI_BASE_URL`. +For Ollama or custom endpoints, supply `baseUrl` and use `ollama-local` as the API key. Provider type is inferred by the runtime from the `provider` field (when present) or from key patterns (see [`MemoriAI.detectProvider`](src/core/MemoriAI.ts:313) and [`Memori.detectProviderType`](src/core/Memori.ts:155)). The `ConfigManager` recognises `DATABASE_URL`, `MEMORI_NAMESPACE`, `MEMORI_AUTO_INGEST`, `MEMORI_CONSCIOUS_INGEST`, `MEMORI_ENABLE_RELATIONSHIP_EXTRACTION`, `OPENAI_API_KEY`, and `OPENAI_BASE_URL`. ## Documentation Map @@ -48,7 +51,7 @@ For Ollama or custom endpoints, supply `baseUrl` and the synthetic API key (`oll - [Getting Started](getting-started.md) – installation, environment setup, and first request. - [Basic Usage](basic-usage.md) – day-to-day patterns with `MemoriAI` plus pointers to advanced workflows. - [Memory Management](core-concepts/memory-management.md) – automatic vs conscious ingestion aligned with `Memori`. - - [Search Strategies](core-concepts/search-strategies.md) – how strategies in `src/core/domain/search` are orchestrated. + - [Search Strategies](core-concepts/search-strategies.md) – overview of implemented search strategies and their use via stable APIs. - [Category Hierarchy](core-concepts/category-hierarchy.md) – hierarchical category management with `CategoryHierarchyManager` and `CategoryMetadataExtractor`. - **Architecture** diff --git a/docs/developer/operations/conscious-mode-and-background-jobs.md b/docs/developer/operations/conscious-mode-and-background-jobs.md new file mode 100644 index 0000000..39358ad --- /dev/null +++ b/docs/developer/operations/conscious-mode-and-background-jobs.md @@ -0,0 +1,253 @@ +# Conscious Mode and Background Jobs + +This guide describes how to operate Memori in conscious mode using only stable, documented APIs. + +It is based on: +- [`Memori`](src/core/Memori.ts:21) +- Conscious ingestion flags managed by [`ConfigManager`](src/core/infrastructure/config/ConfigManager.ts:29) +- Database and state handling in [`DatabaseManager`](src/core/infrastructure/database/DatabaseManager.ts:138) + +Goals: +- Explain what "conscious" mode does. +- Show how to schedule background processing safely. +- Provide guidance for multi-instance deployments. + +Only documented methods are used; direct access to private fields or deep internals is considered unsupported. + +--- + +## 1. What Conscious Mode Does + +Conscious mode is designed for deferred, batch-style processing of important memories. + +When enabled: + +- Conversations are stored immediately. +- Memory processing (promotion, consolidation, etc.) happens asynchronously via: + - `ConsciousAgent` (internal) + - `ConsciousMemoryManager` / related DB managers + +Configuration: + +- Via code (Memori): + +```ts +import { Memori } from 'memorits'; + +const memori = new Memori({ + databaseUrl: 'file:./memori.db', + mode: 'conscious', + namespace: 'assistant-a', +}); + +await memori.enable(); +``` + +- Via environment (ConfigManager): + - `MEMORI_CONSCIOUS_INGEST=true` + - `MEMORI_AUTO_INGEST=false` (if you want pure conscious mode) + +Result: +- `Memori.enable()`: + - Instantiates `ConsciousAgent` when conscious mode is active. + - May perform an initial ingestion run. + - Optionally starts background monitoring if configured. + +Reference: +- [`Memori.enable`](src/core/Memori.ts:174) + +--- + +## 2. Key Public Operations + +### 2.1 initializeConsciousContext() + +```ts +await memori.initializeConsciousContext(); +``` + +Use: +- To bootstrap from existing `conscious-info` memories into short-term context after startup. + +Behavior: +- No-op if: + - Memori is not enabled. + - Conscious mode is not active or `ConsciousAgent` is unavailable. +- Logs errors; does not throw for transient issues. + +Reference: +- [`Memori.initializeConsciousContext`](src/core/Memori.ts:534) + +### 2.2 checkForConsciousContextUpdates() + +```ts +await memori.checkForConsciousContextUpdates(); +``` + +Use: +- To process new conscious memories on demand (e.g., cron/worker). + +Behavior: +- No-op when: + - Memori is not enabled. + - ConsciousAgent is not initialized. +- When active: + - Pulls new `conscious-info` items. + - Applies promotions/updates via internal services. + - Logs any errors; does not throw for normal failures. + +Reference: +- [`Memori.checkForConsciousContextUpdates`](src/core/Memori.ts:514) + +### 2.3 Background Monitoring Controls + +Memori can run its own timer when conscious mode is enabled. + +APIs: + +- `setBackgroundUpdateInterval(intervalMs: number)` + - Updates the interval; restarts monitoring if active. +- `getBackgroundUpdateInterval(): number` +- `isBackgroundMonitoringActive(): boolean` + +References: +- [`Memori.setBackgroundUpdateInterval`](src/core/Memori.ts:617) +- [`Memori.isBackgroundMonitoringActive`](src/core/Memori.ts:641) + +Notes: +- `startBackgroundMonitoring` / `stopBackgroundMonitoring` are internal helpers; use the public interval APIs to influence behavior, not direct timer access. + +--- + +## 3. Recommended Scheduling Patterns + +### 3.1 Single-Instance Deployment + +Simplest approach: + +- Enable conscious mode on your main process: + +```ts +const memori = new Memori({ databaseUrl: 'file:./memori.db', mode: 'conscious' }); +await memori.enable(); +``` + +Options: + +1. Use built-in monitoring: + - Rely on Memori’s internal background monitoring. + - Optionally tune with `setBackgroundUpdateInterval(ms)`. + +2. External scheduler (preferred for explicit control): + - Disable or ignore built-in timer. + - Use a cron/worker to trigger: + +```ts +async function runConsciousTick(memori: Memori) { + await memori.checkForConsciousContextUpdates(); +} + +setInterval(() => runConsciousTick(memori), 60_000); +``` + +Guidance: +- Keep interval >= tens of seconds to avoid thrashing. +- Monitor logs for errors (component `Memori` / `ConsciousAgent`). + +### 3.2 Multi-Instance Deployment + +When running multiple app instances: + +Concerns: +- Avoid duplicated work / race conditions. +- Avoid each instance running its own tight loop by default. + +Recommended patterns: + +1. Dedicated worker instance: + - One Memori instance (or service) runs with: + - `mode: 'conscious'` + - A controlled scheduler calling `checkForConsciousContextUpdates`. + - Other instances: + - Use `automatic` or `manual` modes or `conscious` without scheduling. + - Do not run background updates. + +2. External orchestrator: + - Use a job system (e.g. cron, queue) to call an endpoint that: + - Acquires a simple distributed lock (e.g., DB-based). + - Runs `checkForConsciousContextUpdates` under that lock. + - Ensures only one worker performs conscious updates at a time. + +Avoid: +- Having N identical instances all run background monitoring without coordination. + +--- + +## 4. Namespace and Tenancy Considerations + +Conscious mode respects namespaces via underlying storage: + +- Use distinct `namespace` values per tenant / logical agent. +- Run conscious updates per namespace if needed. + +Common patterns: + +- Single shared DB, multiple namespaces: + - One consolidation/processing worker handling all namespaces. +- Per-tenant Memori instance: + - Conscious scheduling per instance with its own namespace. + +Ensure: +- You pass consistent `namespace` config when constructing Memori/MemoriAI. +- You do not mix unrelated workloads into the same namespace. + +--- + +## 5. Failure Modes and Observability + +Conscious processing methods are defensive: + +- `initializeConsciousContext`: + - Logs and returns on error. +- `checkForConsciousContextUpdates`: + - Catches and logs errors; failures do not crash the process. + +Your responsibilities: + +- Monitor logs: + - Filter by `component: 'Memori'` and `component: 'ConsciousAgent'`. +- Treat repeated failures as signals: + - Schema issues + - Permission problems + - Misconfiguration (e.g., wrong database path) + +Recommended: +- Wire logs into your existing observability stack. +- Add alerts for: + - Frequent conscious update failures. + - Unexpectedly long processing times (from your own measurements). + +--- + +## 6. Stability Notes + +Stable and recommended: +- `mode: 'conscious'` via config or env. +- `Memori.initializeConsciousContext()` +- `Memori.checkForConsciousContextUpdates()` +- `Memori.setBackgroundUpdateInterval()` +- `Memori.getBackgroundUpdateInterval()` +- `Memori.isBackgroundMonitoringActive()` + +Advanced but supported: +- Running a dedicated worker / scheduler that calls `checkForConsciousContextUpdates` across namespaces with your own locking or coordination. + +Internal / not guaranteed: +- Direct access to: + - `ConsciousAgent` + - `ConsciousMemoryManager` + - Private timers or internal scheduling methods +- Any `memori['consciousAgent']` or `memori['dbManager']` style access: + - Treat as unsupported; wrap at your own risk. + +This guide is intended to reflect the real conscious-mode behavior. If you observe mismatches between documented and actual runtime behavior, treat that as a documentation or implementation bug to be resolved. \ No newline at end of file diff --git a/docs/developer/providers/overview.md b/docs/developer/providers/overview.md new file mode 100644 index 0000000..b38a63b --- /dev/null +++ b/docs/developer/providers/overview.md @@ -0,0 +1,290 @@ +# Providers Overview + +This guide documents how Memorits integrates with LLM providers and how to configure them correctly. + +It is grounded in: +- [`MemoriAI`](src/core/MemoriAI.ts:37) +- [`Memori`](src/core/Memori.ts:21) +- [`ConfigManager`](src/core/infrastructure/config/ConfigManager.ts:29) +- Provider infrastructure under `src/core/infrastructure/providers/` + +Goals: +- Explain provider detection for MemoriAI and Memori. +- Describe Memori’s dual-provider architecture. +- Provide concrete configuration examples for OpenAI, Anthropic, and Ollama-like setups. + +--- + +## 1. Provider Detection + +### 1.1 MemoriAI + +MemoriAI chooses the provider via: + +1. Explicit `provider` in `MemoriAIConfig`: + - `'openai'` → `ProviderType.OPENAI` + - `'anthropic'` → `ProviderType.ANTHROPIC` + - `'ollama'` → `ProviderType.OLLAMA` + +2. Otherwise by API key / pattern: + - `sk-ant-*` → Anthropic + - `sk-*` (length > 20) → OpenAI + - `ollama-local` → Ollama + +3. Fallback: + - Defaults to OpenAI if nothing else matches. + +Reference: +- [`MemoriAI.detectProvider`](src/core/MemoriAI.ts:313) + +### 1.2 Memori + +Memori performs simpler detection for its internal providers: + +- `detectProviderType(config: IProviderConfig)`: + - `apiKey` starting with `sk-ant-` → Anthropic. + - `apiKey` starting with `sk-` and long enough → OpenAI. + - `apiKey === 'ollama-local'` → Ollama. + - Default → OpenAI. + +Reference: +- [`Memori.detectProviderType`](src/core/Memori.ts:155) + +Note: +- Memori primarily relies on already-sanitized config from `ConfigManager` and the supplied overrides. + +### 1.3 ConfigManager and Environment + +`ConfigManager.loadConfig()` is responsible for env-driven defaults: + +- Required / used environment variables: + - `DATABASE_URL` / `MEMORI_DATABASE_URL` (normalized to `databaseUrl`) + - `MEMORI_NAMESPACE` + - `MEMORI_CONSCIOUS_INGEST` + - `MEMORI_AUTO_INGEST` + - `MEMORI_ENABLE_RELATIONSHIP_EXTRACTION` + - `MEMORI_MODEL` + - `OPENAI_API_KEY` + - `OPENAI_BASE_URL` (used for custom / Ollama-like endpoints) + +Key behavior: +- If no valid `OPENAI_API_KEY` and no `OPENAI_BASE_URL`: + - Throws a `ValidationError`. +- If `OPENAI_BASE_URL` is set without a usable key: + - Sets `apiKey = 'ollama-local'` as a synthetic key. +- All values are sanitized; bad inputs raise `SanitizationError` / `ValidationError`. + +Reference: +- [`ConfigManager.loadConfig`](src/core/infrastructure/config/ConfigManager.ts:29) + +--- + +## 2. Memori Dual-Provider Architecture + +Memori uses a dual-provider design to avoid recursive memory processing: + +1. User Provider (`userProvider`) + - A `MemoryCapableProvider`. + - Handles user-facing operations (chat, embeddings). + - Initialized with memory features enabled/disabled based on config. + - Connected to the shared `DatabaseManager`. + +2. Memory Provider (`memoryProvider`) + - An `ILLMProvider`. + - Used internally by `MemoryAgent` for analysis. + - Initialized with memory processing disabled to prevent recursion. + +Flow: +- `initializeProvider()`: + - Builds `providerConfig` (performance + memory features). + - Detects provider type (OpenAI/Anthropic/Ollama). + - Constructs `userProvider` as `MemoryCapableProvider`. + - Constructs `memoryProvider` for analysis with memory disabled. + - Creates `MemoryAgent(memoryProvider, dbManager)`. + +Reference: +- [`Memori.initializeProvider`](src/core/Memori.ts:72) + +Implications: +- You can: + - Use MemoriAI or Memori with a single API key/baseUrl. + - Let the library manage separate analysis vs user-facing traffic. +- Avoid: + - Manually constructing providers that bypass this pattern unless you know the consequences. + +--- + +## 3. Configuration Patterns by Provider + +Below are concrete, code-accurate examples using stable surfaces. + +### 3.1 OpenAI + +Scenario: +- Using OpenAI for both user chat and memory analysis. + +Environment: + +```bash +export DATABASE_URL="file:./memori.db" +export OPENAI_API_KEY="sk-your-openai-key" +``` + +Using MemoriAI: + +```ts +import { MemoriAI } from 'memorits'; + +const ai = new MemoriAI({ + databaseUrl: 'file:./memori.db', + apiKey: process.env.OPENAI_API_KEY!, + mode: 'automatic', + namespace: 'support-bot', +}); + +const reply = await ai.chat({ + messages: [{ role: 'user', content: 'Remember that invoices go out on the 5th.' }], +}); +``` + +Using Memori directly: + +```ts +import { Memori } from 'memorits'; + +const memori = new Memori({ + databaseUrl: 'file:./memori.db', + apiKey: process.env.OPENAI_API_KEY!, + mode: 'automatic', + namespace: 'support-bot', +}); + +await memori.enable(); +``` + +Notes: +- No custom `provider` value is required; detection via `apiKey` is sufficient. + +--- + +### 3.2 Anthropic + +Scenario: +- Using Anthropic for both user chat and memory analysis. + +Environment: + +```bash +export DATABASE_URL="file:./memori.db" +export OPENAI_API_KEY="sk-ant-your-anthropic-key" +``` + +(Anthropic key is read via `OPENAI_API_KEY` in current ConfigManager wiring; detection is based on `sk-ant-` prefix.) + +MemoriAI: + +```ts +const ai = new MemoriAI({ + databaseUrl: 'file:./memori.db', + apiKey: process.env.OPENAI_API_KEY!, + // Optional explicit hint: + provider: 'anthropic', +}); +``` + +Memori: + +```ts +const memori = new Memori({ + databaseUrl: 'file:./memori.db', + apiKey: process.env.OPENAI_API_KEY!, +}); +await memori.enable(); +``` + +Notes: +- Prefix-based detection will route to Anthropic provider. +- Consider using an Anthropic-specific env var in your app code and mapping it to `apiKey` explicitly for clarity. + +--- + +### 3.3 Ollama / Custom Base URL + +Scenario: +- Using an Ollama-compatible endpoint locally. + +Environment: + +```bash +export DATABASE_URL="file:./memori.db" +export OPENAI_BASE_URL="http://localhost:11434/v1" +``` + +Configuration rules: +- If `OPENAI_BASE_URL` is set and no valid key: + - `ConfigManager` sets `apiKey = 'ollama-local'`. +- Memori’s detection: + - `ollama-local` → `ProviderType.OLLAMA`. + +MemoriAI: + +```ts +const ai = new MemoriAI({ + databaseUrl: 'file:./memori.db', + apiKey: 'ollama-local', + baseUrl: process.env.OPENAI_BASE_URL, + provider: 'ollama', // explicit and clear + mode: 'automatic', +}); +``` + +Memori: + +```ts +const memori = new Memori({ + databaseUrl: 'file:./memori.db', + apiKey: 'ollama-local', + baseUrl: process.env.OPENAI_BASE_URL, + mode: 'automatic', +}); +await memori.enable(); +``` + +Notes: +- This keeps configuration explicit and aligns with actual detection logic. +- For non-Ollama custom endpoints that are OpenAI-compatible, use an appropriate key and URL together. + +--- + +## 4. Recommended Patterns + +- Prefer MemoriAI for: + - Simple “drop-in” usage: chat + memory + search via one object. +- Prefer Memori for: + - Advanced control over ingestion, search strategies, consolidation, and operations. + +Best practices: +- Use explicit env vars and map them into config, rather than relying solely on implicit detection. +- Keep `namespace` consistent per tenant/application. +- Use a single shared `databaseUrl` + multiple namespaces when you want shared infrastructure and isolated memory. + +--- + +## 5. Stability Notes + +Public and supported: +- Provider selection via: + - `MemoriAI` config (`provider`, `apiKey`, `baseUrl`, `mode`). + - `Memori` config + `ConfigManager` env handling. +- Dual-provider design of Memori (user vs memory provider) as documented here. +- Using OpenAI, Anthropic, and Ollama-like endpoints via the patterns above. + +Advanced but supported: +- Tuning provider performance/memory features through `IProviderConfig.features` when using low-level provider factories. +- Sharing a database between multiple Memori/MemoriAI instances for multi-tenant cases (with distinct namespaces). + +Internal: +- Direct imports from `src/core/infrastructure/providers/**` and manual provider wiring. +- Reaching into private properties (e.g. `memori['userProvider']`) is unsupported and subject to change. + +If you find discrepancies between this document and the actual provider-related code, treat that as a documentation bug. \ No newline at end of file