Technical Specification

A.I.M. Technical Specification

This document provides the literal, metaphor-free engineering specifications for the A.I.M. exoskeleton.

1. Database Schema (engram.db)

The core memory layer is a local SQLite database utilizing parameterized queries to prevent SQL injection. It implements Hybrid Retrieval (Semantic Cosine Similarity + FTS5 BM25).

sessions Table

Tracks ingestion sources and provides chronological metadata.

• id (TEXT PRIMARY KEY)
• filename (TEXT NOT NULL)
• mtime (REAL NOT NULL)
• indexed_at (TIMESTAMP DEFAULT CURRENT_TIMESTAMP)

fragments Table

Stores the actual semantic chunks.

• id (INTEGER PRIMARY KEY AUTOINCREMENT)
• session_id (TEXT NOT NULL) - Foreign Key to sessions(id) ON DELETE CASCADE
• type (TEXT NOT NULL) - e.g., 'foundation_knowledge', 'expert_knowledge', 'user_prompt', 'model_response'
• content (TEXT NOT NULL)
• timestamp (TEXT)
• embedding (BLOB) - 768-dimensional float array packed via struct (Nomic standard).
• metadata (TEXT) - JSON string containing tool call arguments or routing data.

fragments_fts (Virtual Table)

An FTS5 virtual table synchronized via 3 SQLite triggers (AFTER INSERT, AFTER DELETE, AFTER UPDATE) on the fragments table to provide zero-latency exact-match keyword search (BM25 algorithm).

2. Token Economics & Benchmarks

The distillation pipeline operates to drastically reduce token payload before semantic embedding.

Empirical Methodology

The extract_signal.py script targets massive raw JSON session files generated by the CLI. It physically strips out all ANSI color codes, verbatim stdout dumps, and system metadata, leaving only the "Signal Skeleton" (Intents, Tool Calls, and Outputs).

Variance Data (Real-World Samples)

Compression ratios vary based on the ratio of "terminal noise" to "architectural logic" in a given session.

• Sample A (Heavy Terminal Execution - 50+ Turns):
  • Raw Payload: 8.76 MB
  • Signal Skeleton: 1.39 MB
  • Result: 6.44x Compression
• Sample B (Standard Debugging Session):
  • Raw Payload: 270.70 KB
  • Signal Skeleton: 62.98 KB
  • Result: 4.30x Compression
• Conclusion: The pipeline reliably achieves between a 4.3x and 8.5x reduction in token weight depending on task verbosity, allowing secondary distillation models (e.g., Gemini Flash, Ollama) to process long-term memory for fractions of a cent.

Zero-Token Filtering:

• extract_signal.py parses .gemini/tmp/.../chats/*.json.
• It strips all conversational filler, tool arguments, tool results, and massive JSON structures.
• The Signal Skeleton: Leaves only the high-level intent (role: user) and the technical output (role: model/gemini). This reduces context weight by up to 85% before the LLM ever sees it.

3. The DataJack Swarm (Knowledge Cartridges)

3.1 Cartridge Format (.engram) Standard ZIP archives containing:

• metadata.json (Title, Author, Target Tiers, Hash)
• chunks/*.jsonl (Pre-embedded text fragments and FTS5 data)

3.2 Injection Pipeline When aim jack-in <file> is executed:

1. Security Sandbox: A.I.M. parses the .jsonl files natively. It does not execute raw SQL scripts.
2. Parameterized Insertion: Fragments are loaded via sqlite3 using explicit parameterization to prevent SQL injection: INSERT INTO fragments (session_id, type, content, timestamp, embedding, metadata) VALUES (?, ?, ?, ?, ?, ?)
3. Operator Consent: The operator is presented with the metadata.json details and must explicitly type y to execute the database write.

4. Source Control & Branching

A.I.M. uses Timestamped Archive Branching (archive-<branch>-<datetime>) instead of Git Tags during the Phase Protocol. This is a deliberate, non-standard design choice. AI CLI agents struggle to navigate detached HEAD states and git reflog recovery. Explicit archival branches provide safe, verifiable "Save States" that an AI can easily list, checkout, and restore from without breaking the local git tree.