fix: improve async batch retain with large payloads (#366)

* fix: improve async batch retain with large payloads

* fix: improve async batch retain with large payloads

* api

* api

* api

* api

* api

* Clean up perf benchmark: keep only Python files

- Remove README.md and PERFORMANCE_FINDINGS.md
- Remove results/ JSON files (gitignored)
- Remove test_data/ directory
- Keep only __init__.py and retain_perf.py

* docs: explain automatic batch optimization for async retain

- Add section explaining Hindsight automatically handles batch sizing
- Users don't need to manually tune batch sizes with async mode
- Hindsight splits large batches (>10k tokens) into optimized sub-batches
- Include example showing best practices

* docs: remove emojis and code example from performance page

* fix: correct OperationDetails type to match API response

- Change optional fields to use | null instead of ?
- Fixes TypeScript compilation error in control plane build

* fix: use discriminated union for OperationDetails type

- Support both success and error states properly
- Fixes TypeScript error when setting error state

* fix: use unique document_ids in batch retain examples

- Each item in a batch must have unique document_id
- Update both Python and JavaScript examples
- Fixes test-doc-examples CI failure

* chore: trigger CI

* fix: test mocking and duplicate document_ids in examples

- Mock _get_pool() in test_async_retain_tags.py to avoid _initialized error
- Set _initialized = True on mocked MemoryEngine instances
- Fix duplicate document_ids in retain.py and retain.mjs examples

* fix: properly mock async pool/connection and fix more duplicate document_ids

- Use AsyncMock for pool.acquire() to fix 'can't be used in await' error
- Fix duplicate document_ids in retain-async examples (retain.py and retain.mjs)
- Remove batch-level document_id parameter that caused duplicates

* ci: collect all doc example failures and show summary

- Run all Python/Node.js/CLI examples regardless of individual failures
- Collect failure list and display summary at the end
- Show pass/fail count and list of failed files
- Exit with failure only after running all examples

* refactor: extract doc example testing to standalone script

- Create scripts/test-doc-examples.sh to run all examples
- Collects logs of failed examples separately
- Shows full error logs only for failures at the end
- Clean summary with pass/fail counts
- Proper exit codes
- Replaces inline bash in CI workflow

* fix: doc examples - duplicate document_ids and error handling

- retain.py: move document_id to item level to avoid duplicates
- documents.mjs: add error handling for getDocument to show clear error message

* fix: update tests for duplicate document_id validation

- test_async_retain_tags: verify operation structure instead of exact UUID
- test_delete_bank: use unique document_ids (team-doc-1, team-doc-2)

Author: nicoloboschi

.github/workflows/test.yml

@@ -941,30 +941,11 @@ jobs:
           sleep 1
         done
 
-    - name: Run Python doc examples
-      working-directory: ./hindsight-clients/python
-      run: |
-        for f in ../../hindsight-docs/examples/api/*.py; do
-          echo "Running $f..."
-          uv run python "$f"
-        done
-
-    - name: Run Node.js doc examples
-      run: |
-        for f in hindsight-docs/examples/api/*.mjs; do
-          echo "Running $f..."
-          node "$f"
-        done
-
     - name: Configure CLI
       run: hindsight configure --api-url http://localhost:8888
 
-    - name: Run CLI doc examples
-      run: |
-        for f in hindsight-docs/examples/api/*.sh; do
-          echo "Running $f..."
-          bash "$f"
-        done
+    - name: Run all doc examples
+      run: ./scripts/test-doc-examples.sh
 
     - name: Show API server logs
       if: always()

.gitignore

@@ -46,6 +46,7 @@ hindsight-docs/static/llms-full.txt
 hindsight-dev/benchmarks/locomo/results/
 hindsight-dev/benchmarks/longmemeval/results/
 hindsight-dev/benchmarks/consolidation/results/
+hindsight-dev/benchmarks/perf/results/
 benchmarks/results/
 hindsight-cli/target
 hindsight-clients/rust/target

CLAUDE.md

@@ -57,8 +57,15 @@ cd hindsight-control-plane && npm run dev
 
 ### Benchmarks
 ```bash
+# Accuracy benchmarks
 ./scripts/benchmarks/run-longmemeval.sh
 ./scripts/benchmarks/run-locomo.sh
+
+# Performance benchmarks
+./scripts/benchmarks/run-consolidation.sh
+./scripts/benchmarks/run-retain-perf.sh --document <path>  # Requires API server running
+
+# Results viewer
 ./scripts/benchmarks/start-visualizer.sh  # View results at localhost:8001
 ```
 

hindsight-api/hindsight_api/alembic/versions/y0t1u2v3w4x5_add_result_metadata_gin_index.py

@@ -0,0 +1,49 @@
+"""Add GIN index on async_operations.result_metadata for parent_operation_id queries
+
+Revision ID: y0t1u2v3w4x5
+Revises: x9s0t1u2v3w4
+Create Date: 2026-02-13
+
+This migration adds a GIN index on the result_metadata JSONB column in the
+async_operations table to support efficient queries for child operations by
+parent_operation_id.
+
+The index enables fast lookups when querying for child operations:
+  SELECT * FROM async_operations
+  WHERE result_metadata::jsonb @> '{"parent_operation_id": "uuid"}'::jsonb
+"""
+
+from collections.abc import Sequence
+
+from alembic import context, op
+
+revision: str = "y0t1u2v3w4x5"
+down_revision: str | Sequence[str] | None = "x9s0t1u2v3w4"
+branch_labels: str | Sequence[str] | None = None
+depends_on: str | Sequence[str] | None = None
+
+
+def _get_schema_prefix() -> str:
+    """Get schema prefix for table names (required for multi-tenant support)."""
+    schema = context.config.get_main_option("target_schema")
+    return f'"{schema}".' if schema else ""
+
+
+def upgrade() -> None:
+    """Add GIN index on result_metadata for efficient parent_operation_id queries."""
+    schema = _get_schema_prefix()
+
+    # Add GIN index for JSONB containment queries (@> operator)
+    op.execute(f"""
+        CREATE INDEX idx_async_operations_result_metadata
+        ON {schema}async_operations
+        USING gin(result_metadata)
+    """)
+
+
+def downgrade() -> None:
+    """Remove GIN index on result_metadata."""
+    schema = _get_schema_prefix()
+
+    # Drop index
+    op.execute(f"DROP INDEX IF EXISTS {schema}idx_async_operations_result_metadata")

hindsight-api/hindsight_api/api/http.py

@@ -1357,6 +1357,16 @@ class CancelOperationResponse(BaseModel):
     operation_id: str
 
 
+class ChildOperationStatus(BaseModel):
+    """Status of a child operation (for batch operations)."""
+
+    operation_id: str
+    status: str
+    sub_batch_index: int | None = None
+    items_count: int | None = None
+    error_message: str | None = None
+
+
 class OperationStatusResponse(BaseModel):
     """Response model for getting a single operation status."""
 
@@ -1381,6 +1391,13 @@ class OperationStatusResponse(BaseModel):
     updated_at: str | None = None
     completed_at: str | None = None
     error_message: str | None = None
+    result_metadata: dict[str, Any] | None = Field(
+        default=None,
+        description="Internal metadata for debugging. Structure may change without notice. Not for production use.",
+    )
+    child_operations: list[ChildOperationStatus] | None = Field(
+        default=None, description="Child operations for batch operations (if applicable)"
+    )
 
 
 class AsyncOperationSubmitResponse(BaseModel):

hindsight-api/hindsight_api/config.py

@@ -250,6 +250,7 @@ def normalize_config_dict(config: dict[str, Any]) -> dict[str, Any]:
 ENV_RETAIN_EXTRACT_CAUSAL_LINKS = "HINDSIGHT_API_RETAIN_EXTRACT_CAUSAL_LINKS"
 ENV_RETAIN_EXTRACTION_MODE = "HINDSIGHT_API_RETAIN_EXTRACTION_MODE"
 ENV_RETAIN_CUSTOM_INSTRUCTIONS = "HINDSIGHT_API_RETAIN_CUSTOM_INSTRUCTIONS"
+ENV_RETAIN_BATCH_TOKENS = "HINDSIGHT_API_RETAIN_BATCH_TOKENS"
 
 # Observations settings (consolidated knowledge from facts)
 ENV_ENABLE_OBSERVATIONS = "HINDSIGHT_API_ENABLE_OBSERVATIONS"
@@ -371,6 +372,7 @@ def normalize_config_dict(config: dict[str, Any]) -> dict[str, Any]:
 DEFAULT_RETAIN_EXTRACTION_MODE = "concise"  # Extraction mode: "concise", "verbose", or "custom"
 RETAIN_EXTRACTION_MODES = ("concise", "verbose", "custom")  # Allowed extraction modes
 DEFAULT_RETAIN_CUSTOM_INSTRUCTIONS = None  # Custom extraction guidelines (only used when mode="custom")
+DEFAULT_RETAIN_BATCH_TOKENS = 10_000  # ~40KB of text  # Max chars per sub-batch for async retain auto-splitting
 
 # Observations defaults (consolidated knowledge from facts)
 DEFAULT_ENABLE_OBSERVATIONS = True  # Observations enabled by default
@@ -590,6 +592,7 @@ class HindsightConfig:
     retain_extract_causal_links: bool
     retain_extraction_mode: str
     retain_custom_instructions: str | None
+    retain_batch_tokens: int
 
     # Observations settings (consolidated knowledge from facts)
     enable_observations: bool
@@ -939,6 +942,7 @@ def from_env(cls) -> "HindsightConfig":
                 os.getenv(ENV_RETAIN_EXTRACTION_MODE, DEFAULT_RETAIN_EXTRACTION_MODE)
             ),
             retain_custom_instructions=os.getenv(ENV_RETAIN_CUSTOM_INSTRUCTIONS) or DEFAULT_RETAIN_CUSTOM_INSTRUCTIONS,
+            retain_batch_tokens=int(os.getenv(ENV_RETAIN_BATCH_TOKENS, str(DEFAULT_RETAIN_BATCH_TOKENS))),
             # Observations settings (consolidated knowledge from facts)
             enable_observations=os.getenv(ENV_ENABLE_OBSERVATIONS, str(DEFAULT_ENABLE_OBSERVATIONS)).lower() == "true",
             consolidation_batch_size=int(