Add session persistence with multipath exploration

[justin4957]

Summary

Implements MVP of session persistence system with ETS-based hot storage, file persistence, and session forking for multipath exploration.

Closes #5

Features Implemented

🔥 ETS Hot Storage

• Sub-millisecond session access with O(1) lookups
• Four specialized ETS tables:
  • sessions - Main session storage
  • indexes - Tag and metadata indexing
  • forks - Parent-child relationships
  • access_tracker - Access pattern monitoring
• Concurrent read optimization with read_concurrency: true

💾 File Persistence

• save_session_to_disk/1 - Save to ~/.multi_agent_coder/sessions/
• load_session_from_disk/1 - Load from disk into ETS
• export_session/2 - Export to JSON (portable format)
• import_session/1 - Import from JSON

🌲 Session Forking (Multipath Exploration)

• fork_session/2 - Branch conversation at any message
• get_session_forks/1 - List all child sessions
• get_session_parent/1 - Navigate up the session tree
• Graph-ready structure for complex exploration patterns

🔍 Search & Discovery

• find_sessions_by_tag/1 - Tag-based search
• find_sessions_by_date_range/2 - Time-based queries
• list_sessions/0 - List all sessions
• Rich metadata support for custom filtering

📊 Usage Tracking

• Token consumption per session
• Cost estimation across providers
• Access count and last accessed tracking
• Provider usage aggregation

Architecture

MultiAgentCoder.Session.Storage (GenServer)
├── ETS Tables (4)
│   ├── :mac_sessions         # Main storage
│   ├── :mac_session_indexes  # Tag/metadata indexing
│   ├── :mac_session_forks    # Parent-child relationships
│   └── :mac_access_tracker   # Access patterns
├── File Persistence
│   └── ~/.multi_agent_coder/sessions/*.json
└── Data Structures
    ├── Session (id, messages, metadata, providers_used, etc.)
    └── Message (id, role, content, provider, tokens, etc.)

Implementation Details

Session Data Structure

%Session{
  id: "session_1_1234567890",
  parent_id: nil,                    # For forked sessions
  fork_point: nil,                   # Message index of fork
  created_at: ~U[2025-10-14 08:00:00Z],
  last_accessed_at: ~U[2025-10-14 10:30:00Z],
  access_count: 42,
  messages: [...],                   # Full conversation history
  metadata: %{tags: ["feature"], description: "..."},
  providers_used: [:openai, :anthropic],
  total_tokens: 1500,
  estimated_cost: 0.045,
  retention_policy: :standard
}

Multipath Example

# Main conversation
{:ok, session_id} = Storage.create_session(%{tags: ["caching"]})
Storage.add_message(session_id, %{role: :user, content: "I need a caching layer"})

# Fork to explore ETS approach
{:ok, fork1} = Storage.fork_session(session_id, 
  at_message: 1,
  metadata: %{fork_reason: "ETS approach"}
)

# Fork to explore Redis approach
{:ok, fork2} = Storage.fork_session(session_id,
  at_message: 1, 
  metadata: %{fork_reason: "Redis approach"}
)

# Continue independently, then compare
{:ok, forks} = Storage.get_session_forks(session_id)

Testing

Comprehensive test coverage with 13 tests:

✅ Session management (create, retrieve, list, find by tag, delete)
✅ Message management (add messages, track providers, token counting)
✅ Session forking (fork creation, relationships, parent tracking)
✅ Persistence (save/load from disk, export/import JSON)
✅ Statistics (storage stats and memory usage)

All tests passing:

mix test test/multi_agent_coder/session/storage_test.exs

Future Compatibility

The implementation is designed to be compatible with Grapple integration:

• Graph Structure: Sessions and forks form a natural graph
• Tiered Storage: Easy migration to ETS → Mnesia → DETS tiers
• Query Patterns: Tag-based indexing maps to graph queries
• Scalability: Ready for distributed session storage

This allows for future enhancements:

• Distributed session replication
• Complex graph queries across session trees
• Advanced analytics on conversation patterns
• Session clustering and recommendation

Documentation

Added comprehensive documentation to README including:

• Core features overview
• Basic session operations with code examples
• Multipath exploration patterns
• Session tree navigation
• Search and discovery API
• Future Grapple integration notes

Files Changed

• lib/multi_agent_coder/session/storage.ex - Main storage module (414 lines)
• lib/multi_agent_coder/session/storage/session.ex - Session data structure
• lib/multi_agent_coder/session/storage/message.ex - Message data structure
• lib/multi_agent_coder/application.ex - Added Storage to supervision tree
• test/multi_agent_coder/session/storage_test.exs - Comprehensive tests (217 lines)
• README.md - Added "Session Persistence and Multipath Exploration" section
• .gitignore - Fixed to not block lib/multi_agent_coder directory

Test Plan

• All tests passing (13/13)
• Session creation and retrieval
• Message addition and token tracking
• Session forking and tree navigation
• File persistence (save/load)
• JSON export/import
• Tag-based search
• Storage statistics
• CI tests passing (will monitor)

🤖 Generated with Claude Code

[justin4957]