Skip to content

TannerBurns/secular

Repository files navigation

Secular - AI Agent Framework

A comprehensive framework for creating AI agents with multi-agent coordination, memory systems, tool management, consciousness for planning and reflection, and runtime abstraction for flexible execution.

Features

  • Multi-Agent Coordination: Create and coordinate multiple AI agents working together
  • YAML Configuration: Define agents declaratively with YAML files or programmatically with code
  • Advanced Memory Systems: Hot cache (Redis), persistent storage (Postgres), and semantic search (pgvector)
  • Tool Management: Flexible tool registration with permissions, async/parallel execution, and metrics
  • Consciousness System: Planning, task tracking, goal management, and iterative reflection
  • Context Management: Intelligent message history pruning and context window management
  • Usage Tracking: Comprehensive tracking of LLM calls, tokens, costs, and agent performance
  • Multi-Tenancy: First-class support for tenants, users, departments, and sessions
  • Runtime Abstraction: Extensible runtime system with local execution
  • Type Safety: Strict type checking with mypy and Pydantic models everywhere
  • Comprehensive Testing: 596 tests with 91% coverage and mocked LLM calls

Tech Stack

  • Python 3.13+
  • uv - Fast Python package manager
  • PostgreSQL with pgvector - Persistent storage and semantic search
  • Redis - Hot cache
  • LiteLLM - Unified LLM interface (OpenAI, Anthropic, etc.)
  • SQLAlchemy 2.0 - Async ORM
  • Alembic - Database migrations
  • Pydantic - Data validation and settings
  • Dynaconf - Configuration management
  • Pytest - Testing framework

Installation

Quick Start (No Database Required)

Get started with Secular in under a minute using in-memory mode:

  1. Install Python 3.13+ if you don't have it

  2. Install uv (fast Python package manager):

curl -LsSf https://astral.sh/uv/install.sh | sh
  1. Install the package (choose one method):

Option A - Install directly from GitHub:

pip install git+https://github.com/your-username/secular.git

Option B - Clone and install with uv (for development):

git clone https://github.com/your-username/secular.git
cd secular
uv sync
  1. Set your API key:
export OPENAI_API_KEY="sk-your-key"
# or
export ANTHROPIC_API_KEY="sk-ant-your-key"
  1. Try your first agent (create a file quick_start.py):
import asyncio
from secular import AgentBuilder
from secular.schemas.agent import AgentInput
from secular.schemas.tenancy import TenancyContext
from secular.tools.domains.calculator import CalculatorTool

async def main():
    # Create agent with builder pattern (no database needed!)
    agent = await (
        AgentBuilder()
        .with_name("QuickStartAgent")
        .with_system_prompt("You are a helpful assistant.")
        .with_max_iterations(5)
        .with_tools([CalculatorTool()])
        .build()
    )
    
    # Run agent
    output = await agent.run(
        AgentInput(
            content="What is 127 * 43?",
            context=TenancyContext(),
        )
    )
    
    print(f"Agent: {output.content}")

if __name__ == "__main__":
    asyncio.run(main())
  1. Run it:
uv run python quick_start.py

Alternative: YAML Configuration

Prefer configuration over code? Define agents in YAML:

agents.yaml:

my_assistant:
  name: "My Assistant"
  system_prompt: "You are a helpful assistant."
  model: "gpt-4"
  temperature: 0.7
  tools:
    - "secular.tools.domains.calculator.AddTool"

Load and use:

from secular.loader import load_agent_from_yaml

agent = await load_agent_from_yaml("my_assistant")
output = await agent.run(AgentInput(content="What is 5 + 3?"))

See YAML Configuration Guide for complete documentation.

That's it! You now have a working AI agent with tool access and memory, all without any database setup.

Production Setup (PostgreSQL + Redis)

For production deployments with persistent storage:

Prerequisites:

  • PostgreSQL 16+ with pgvector extension
  • Redis 7+

Setup:

  1. Start database services:
docker-compose up -d
  1. Run database migrations:
uv run alembic upgrade head
  1. Configure secrets (optional - can use environment variables):
cp .settings.secrets.yaml.example .settings.secrets.yaml

Edit .settings.secrets.yaml with your API keys:

default:
  llm:
    openai_api_key: "sk-your-openai-api-key-here"
    anthropic_api_key: "sk-ant-your-anthropic-api-key-here"
  1. Update your code to use database mode:
from secular import AgentBuilder
from secular.db.session import get_session
from secular.interactor.embeddings import LiteLLMEmbeddingInteractor

async def main():
    async for session in get_session():
        embedding_interactor = LiteLLMEmbeddingInteractor()
        
        # Create agent with database backing
        agent = await (
            AgentBuilder()
            .with_name("MyAgent")
            .with_system_prompt("You are a helpful assistant")
            .with_session(session)
            .with_embedding_interactor(embedding_interactor)
            .build()
        )
        break

See the Getting Started Guide for detailed examples and Deployment Guide for production best practices.

Getting Started

For comprehensive guides on understanding and creating agents with this framework, please refer to the docs/ directory. The documentation includes:

  • Getting Started Guide
  • Creating Your First Agent
  • Multi-Agent Coordination
  • Memory Systems Usage
  • Tool Development
  • Architecture Deep Dive

Configuration Options

In-Memory Mode for Testing

Secular supports in-memory backends for all memory systems, allowing you to test without requiring external services like PostgreSQL or Redis. This is ideal for unit tests, CI/CD pipelines, and local development.

Using In-Memory Mode

from secular import AgentBuilder
from secular.schemas.agent import AgentInput
from secular.schemas.tenancy import TenancyContext

# Create an agent with all in-memory backends
agent = await (
    AgentBuilder()
    .with_name("test-agent")
    .with_system_prompt("You are a helpful assistant")
    .build()
)

# Run the agent
context = TenancyContext()
agent_input = AgentInput(
    content="Hello, how are you?",
    context=context,
)

result = await agent.run(agent_input)
print(result.content)

Benefits:

  • No external dependencies (PostgreSQL, Redis)
  • Fast test execution
  • Parallel test execution without conflicts
  • Easy CI/CD integration
  • Same API as database-backed mode

Using Your Own Database

Secular allows you to bring your own database configuration, making it easy to integrate with existing infrastructure or use custom connection pools.

Option 1: Inject Custom Database Session

from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession, async_sessionmaker
from secular import AgentBuilder

# Create your own database engine with custom configuration
engine = create_async_engine(
    "postgresql+asyncpg://user:password@host:port/database",
    pool_size=20,
    max_overflow=40,
    echo=True,
)

# Create session factory
session_factory = async_sessionmaker(
    bind=engine,
    class_=AsyncSession,
    expire_on_commit=False,
)

# Create agent with your database session
async with session_factory() as session:
    agent = await (
        AgentBuilder()
        .with_name("my-agent")
        .with_system_prompt("You are helpful")
        .with_session(session)
        .build()
    )
    result = await agent.run(agent_input)

Option 2: Set Global Engine

from sqlalchemy.ext.asyncio import create_async_engine
from secular.db.session import set_engine, get_session
from secular import AgentBuilder

# Create and set custom engine globally
custom_engine = create_async_engine(
    "postgresql+asyncpg://user:password@host:port/database",
    pool_size=20,
)

set_engine(custom_engine)

# Now all agents will use this engine
async for session in get_session():
    agent = await (
        AgentBuilder()
        .with_name("my-agent")
        .with_session(session)
        .build()
    )
    break

Option 3: Custom Memory Backends (Mix and Match)

from secular import AgentBuilder
from secular.memory.manager import MemoryManager
from secular.memory.in_memory_cache import InMemoryCache
from secular.memory.persistent import PostgresPersistentMemory
from secular.memory.semantic import PgVectorSemanticMemory

# Mix in-memory cache with database storage
memory_manager = MemoryManager(
    hot_cache=InMemoryCache(),  # Fast in-memory cache
    persistent=PostgresPersistentMemory(session),  # Database persistent storage
    semantic=PgVectorSemanticMemory(session, embedding_interactor),  # pgvector search
)

agent = await (
    AgentBuilder()
    .with_name("my-agent")
    .with_memory_manager(memory_manager)
    .build()
)

Configuration via Environment Variables

You can configure Secular using environment variables:

# Database configuration
export SECULAR_DATABASE__HOST=localhost
export SECULAR_DATABASE__PORT=5432
export SECULAR_DATABASE__USER=myuser
export SECULAR_DATABASE__PASSWORD=mypassword
export SECULAR_DATABASE__DATABASE=mydb

# Or use direct URL
export SECULAR_DATABASE__URL="postgresql+asyncpg://user:password@host:port/db"

# Redis configuration
export SECULAR_REDIS__HOST=localhost
export SECULAR_REDIS__PORT=6379
export SECULAR_REDIS__DB=0

# Memory backend selection
export SECULAR_MEMORY__BACKEND=auto  # Options: database, in_memory, auto

Configuration via settings.yaml

You can also configure via settings.yaml:

database:
  host: localhost
  port: 5432
  user: myuser
  password: mypassword
  database: mydb
  pool_size: 20
  max_overflow: 40

redis:
  host: localhost
  port: 6379
  db: 0

memory:
  backend: auto  # Options: database, in_memory, auto

Architecture

Core Systems

Agent

The agent accepts input and returns output, maintaining state and coordinating with other agents. Agents can have different types (code writing, terminal, coordinator, etc.) and work together in complex relationships. Each agent is configured with a system prompt, execution parameters, and can access different tools based on permissions.

Executor

The agentic loop responsible for running agents and orchestrating all systems. The executor manages:

  • LLM interactions through the interactor
  • Tool execution (synchronous, async, and parallel)
  • Memory operations (read/write/search)
  • Consciousness updates (planning, tracking, reflection)
  • Context management and pruning
  • Usage tracking for monitoring and cost analysis

Interactor

Handles LLM interactions using LiteLLM for model flexibility:

  • Chat: Completions with streaming support for any LLM (OpenAI, Anthropic, etc.)
  • Embeddings: Vector generation (text-embedding-3-small, 1536 dimensions)
  • Function Calling: Structured tool invocation
  • Usage Tracking: Token counting and cost estimation

Tools

Manages tool availability and execution:

  • Permission-Based Access: Read, write, and admin permissions
  • Execution Modes: Synchronous, asynchronous, and parallel execution
  • Domain Organization: Tools grouped by domain (code, terminal, web, etc.)
  • Metrics: Execution time, success rate, and usage tracking
  • Tool Selection: Automatic tool selection based on context and task

Consciousness

Agent's planning and reflection capabilities:

  • Planner: Task decomposition using LLM-powered analysis
  • Tracker: Task and goal progress tracking with state management
  • Reflector: Iterative reflection with decisions (continue/pivot/escalate)

The consciousness system enables agents to:

  • Break down complex goals into manageable tasks
  • Track progress across multiple iterations
  • Reflect on outcomes and adjust strategies
  • Escalate when stuck or goals are blocked

Context Management

Intelligent handling of conversation history and context:

  • Message Pruning: Automatic removal of old messages when approaching token limits
  • Summarization: Optional context compression for long conversations
  • Recent Message Protection: Always preserve recent interactions
  • Token Estimation: Smart tracking of context window usage

Memory

Multi-layered memory system:

  • Hot Cache (Redis): Fast temporary storage with TTL for session data
  • Persistent (Postgres): Long-term key-value storage for important state
  • Semantic (pgvector): Vector embeddings for similarity search across memories

Memory supports:

  • Scoped access (tenant, user, department, session, agent)
  • Automatic expiration and cleanup
  • Cross-agent memory sharing
  • Semantic search for relevant context retrieval

Usage Tracking

Comprehensive monitoring of agent operations:

  • LLM Metrics: Prompt tokens, completion tokens, total tokens, costs
  • Agent Metrics: Iterations, tool calls, execution time, success rate
  • Persistence: All metrics stored in database for analysis
  • Aggregation: Query usage by tenant, user, agent, time period

Runtime Abstraction

Provides an extensible runtime abstraction:

  • Local Runtime: Direct in-process execution for development and production use cases
  • Extensible: Implement custom runtimes for distributed execution patterns

Tenancy

All systems support multi-tenancy with:

  • Tenant: Top-level organization
  • User: User accounts within tenants
  • Department: Organizational structure within tenants
  • Session: Execution sessions for tracking related interactions

Configuration

Configuration is managed through settings.yaml with environment-specific overrides using Dynaconf.

Main Configuration (settings.yaml)

default:
  app_name: secular
  debug: false

  database:
    host: localhost
    port: 5432
    user: secular
    database: secular_dev
    pool_size: 10
    max_overflow: 20

  redis:
    host: localhost
    port: 6379
    db: 0
    decode_responses: true

  llm:
    default_model: anthropic/claude-sonnet-4-5
    default_temperature: 1.0
    default_max_tokens: 32000
    timeout: 120
    base_url: null  # For OpenAI-compatible providers
    embeddings_base_url: null  # If embeddings API is at a different URL

  embeddings:
    model: text-embedding-3-small
    dimension: 1536
    batch_size: 100

  memory:
    hot_cache_ttl: 3600
    persistent_retention_days: 90
    semantic_similarity_threshold: 0.7

  tools:
    max_parallel_executions: 10
    execution_timeout: 300

  consciousness:
    max_plan_depth: 5
    max_tasks_per_plan: 50
    reflection_interval: 10

  context:
    max_messages: 100
    context_window_tokens: 32000
    keep_recent_messages: 10

Secrets Configuration (.settings.secrets.yaml)

default:
  llm:
    openai_api_key: "sk-your-key"
    anthropic_api_key: "sk-ant-your-key"

Configuration Overrides

Override settings using environment variables prefixed with SECULAR_:

export SECULAR_LLM__DEFAULT_MODEL="gpt-4"
export SECULAR_DATABASE__HOST="prod-db.example.com"

Using OpenAI-Compatible Providers

The framework supports OpenAI-compatible AI providers like vLLM, Ollama, and LMStudio through the base_url configuration. This allows you to use locally hosted models or alternative API endpoints.

vLLM Configuration

vLLM is a high-throughput and memory-efficient inference engine for LLMs.

llm:
  base_url: "http://localhost:8000/v1"
  default_model: "openai/meta-llama/Llama-3-8b"
  api_key: "your-api-key"  # Optional, depending on your vLLM setup

Or via environment variables:

export SECULAR_LLM__BASE_URL="http://localhost:8000/v1"
export SECULAR_LLM__DEFAULT_MODEL="openai/meta-llama/Llama-3-8b"

Ollama Configuration

Ollama allows you to run open-source LLMs locally.

llm:
  base_url: "http://localhost:11434"
  default_model: "ollama/llama2"
  # api_key is not required for Ollama

Or via environment variables:

export SECULAR_LLM__BASE_URL="http://localhost:11434"
export SECULAR_LLM__DEFAULT_MODEL="ollama/llama2"

LMStudio Configuration

LMStudio provides a desktop application for running LLMs locally.

llm:
  base_url: "http://localhost:1234/v1"
  default_model: "openai/local-model"
  # api_key is not required for LMStudio

Or via environment variables:

export SECULAR_LLM__BASE_URL="http://localhost:1234/v1"
export SECULAR_LLM__DEFAULT_MODEL="openai/local-model"

Separate Embeddings Endpoint

If your embeddings API is hosted at a different URL than your chat completions API, use embeddings_base_url:

llm:
  base_url: "http://localhost:8000/v1"  # For chat completions
  embeddings_base_url: "http://localhost:8001/v1"  # For embeddings
  default_model: "openai/meta-llama/Llama-3-8b"

embeddings:
  model: "text-embedding-model"

Model Naming Conventions

When using OpenAI-compatible providers with LiteLLM, follow these model naming conventions:

  • vLLM: Use openai/ prefix (e.g., openai/meta-llama/Llama-3-8b)
  • Ollama: Use ollama/ prefix (e.g., ollama/llama2, ollama/mistral)
  • LMStudio: Use openai/ prefix (e.g., openai/local-model)

These prefixes tell LiteLLM which provider adapter to use when making API calls.

Testing

Running Tests

# Run all tests
uv run pytest

# Run only unit tests
uv run pytest -m unit

# Run only integration tests
uv run pytest -m integration

# Run specific test file
uv run pytest tests/unit/test_agent.py

# Run with coverage
uv run pytest --cov=secular --cov-report=html

# Run with verbose output
uv run pytest -v

# Run with specific markers
uv run pytest -m "unit and not slow"

Test Structure

  • Unit Tests (tests/unit/): Fast tests with mocked dependencies
  • Integration Tests (tests/integration/): Tests with real databases but mocked LLM calls

Integration tests use separate database and Redis instances (ports 5433 and 6380) to avoid conflicts with development data.

Coverage Reports

After running tests with coverage, open the HTML report:

open htmlcov/index.html

Development Workflow

Code Quality Tools

Format Code

# Format all files
uv run ruff format

# Format specific directory
uv run ruff format secular/

# Check formatting without modifying
uv run ruff format --check

Lint Code

# Lint all files
uv run ruff check

# Lint with auto-fix
uv run ruff check --fix

# Lint specific files
uv run ruff check secular/agent/

# Show lint rule explanations
uv run ruff check --show-source

Type Checking

# Type check all files
uv run mypy secular

# Type check specific module
uv run mypy secular/agent

# Show error codes
uv run mypy secular --show-error-codes

# Generate coverage report
uv run mypy secular --html-report mypy-report

Security Checks

# Run security audit
uv run bandit -r secular

# Generate baseline (ignore existing issues)
uv run bandit-baseline -r secular

# Check against baseline
uv run bandit -r secular -b bandit-baseline.json

Pre-commit Hooks

Install pre-commit hooks to automatically run checks before commits:

# Install hooks
uv run pre-commit install

# Run manually on all files
uv run pre-commit run --all-files

# Run on staged files
uv run pre-commit run

# Update hook versions
uv run pre-commit autoupdate

Database Migrations

Creating Migrations

# Auto-generate migration from model changes
uv run alembic revision --autogenerate -m "description of changes"

# Create empty migration for manual changes
uv run alembic revision -m "description of changes"

Applying Migrations

# Upgrade to latest
uv run alembic upgrade head

# Upgrade one version
uv run alembic upgrade +1

# Upgrade to specific revision
uv run alembic upgrade <revision_id>

Reverting Migrations

# Downgrade one version
uv run alembic downgrade -1

# Downgrade to specific revision
uv run alembic downgrade <revision_id>

# Downgrade to base (empty database)
uv run alembic downgrade base

Migration History

# Show current revision
uv run alembic current

# Show migration history
uv run alembic history

# Show pending migrations
uv run alembic heads

Debugging

Enable Debug Logging

Set debug mode in settings.yaml or via environment:

development:
  debug: true
  database:
    echo: true  # Log all SQL queries

Or:

export SECULAR_DEBUG=true
export SECULAR_DATABASE__ECHO=true

Interactive Debugging

Use ipdb for interactive debugging:

import ipdb; ipdb.set_trace()

Install if not present:

uv add --dev ipdb

Performance Profiling

Profile Python Code

# Profile a script
python -m cProfile -o output.prof script.py

# Analyze profile
python -m pstats output.prof

Database Query Performance

Enable SQL query logging:

development:
  database:
    echo: true

Monitor slow queries in PostgreSQL:

SELECT * FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;

CI/CD Considerations

For continuous integration, ensure:

  1. Environment Setup: Use test environment configuration
  2. Service Dependencies: Start PostgreSQL and Redis containers
  3. Database Migrations: Run alembic upgrade head before tests
  4. Parallel Testing: Use pytest-xdist for faster test execution
  5. Coverage Requirements: Set minimum coverage thresholds
  6. Linting: Run ruff, mypy, and bandit in CI pipeline

Example CI test command:

# Start services (if not already running)
docker-compose up -d

# Wait for services
sleep 5

# Run tests with coverage
# Tests use tenant-scoped isolation in the same database as dev
# Redis DB is automatically set to 1 in tests (see tests/conftest.py)
uv run pytest --cov=secular --cov-fail-under=80

# Run linters
uv run ruff check
uv run mypy secular
uv run bandit -r secular

Project Structure

secular/
├── secular/              # Main package
│   ├── agent/            # Agent system
│   ├── consciousness/    # Planning, tracking, reflection
│   ├── context/          # Context management
│   ├── db/               # Database utilities and session management
│   ├── executor/         # Agent execution loops
│   ├── interactor/       # LLM interactions (chat, embeddings)
│   ├── memory/           # Memory systems (hot cache, persistent, semantic)
│   ├── models/           # SQLAlchemy models
│   ├── parsing/          # Response parsing
│   ├── runtime/          # Runtime abstraction (local)
│   ├── schemas/          # Pydantic schemas
│   ├── settings/         # Configuration management
│   ├── tools/            # Tool management and execution
│   └── usage/            # Usage tracking
├── tests/                # Test suite
│   ├── unit/             # Unit tests with mocked dependencies
│   ├── integration/      # Integration tests with real databases
│   ├── conftest.py       # Pytest fixtures
│   └── fixtures/         # Test data and fixtures
├── alembic/              # Database migrations
│   └── versions/         # Migration files
├── settings.yaml         # Main configuration
├── .settings.secrets.yaml # Secrets (gitignored)
├── docker-compose.yml    # Development services
├── pyproject.toml        # Python project configuration
└── README.md             # This file

Design Principles

  1. Type Safety: Use Pydantic models everywhere, no getattr/hasattr
  2. Strict Typing: Reduce Any usage, use TypeVar and Protocol
  3. No Lazy Imports: All imports at the top of files
  4. Dependency Injection: Use patterns for easy testing and reuse
  5. Testability: Every piece should be easily testable
  6. Modularity: Keep files under 500 lines, modularize components

Troubleshooting

Database Connection Issues

Problem: could not connect to server: Connection refused

Solutions:

  • Ensure PostgreSQL is running: docker-compose ps
  • Check connection settings in settings.yaml
  • Verify database exists: psql -h localhost -U secular -l
  • Check logs: docker-compose logs postgres

Problem: relation "tenants" does not exist

Solution: Run migrations: uv run alembic upgrade head

Redis Connection Issues

Problem: Error connecting to Redis

Solutions:

  • Ensure Redis is running: docker-compose ps
  • Check connection settings in settings.yaml
  • Test connection: redis-cli -h localhost -p 6379 ping
  • Check logs: docker-compose logs redis

Migration Issues

Problem: Target database is not up to date

Solution: Run migrations: uv run alembic upgrade head

Problem: Can't locate revision identified by '<hash>'

Solutions:

  • Check migration history: uv run alembic history
  • Reset to base and re-run: uv run alembic downgrade base && uv run alembic upgrade head
  • For development databases, consider dropping and recreating

API Key Issues

Problem: AuthenticationError: Invalid API key

Solutions:

  • Verify API key in .settings.secrets.yaml
  • Check environment variables: echo $OPENAI_API_KEY
  • Ensure no extra spaces or quotes in key
  • Verify key is active in provider dashboard

Problem: litellm.exceptions.BadRequestError

Solutions:

  • Check model name format (e.g., anthropic/claude-sonnet-4-5)
  • Verify model exists and you have access
  • Check rate limits and quota

Test Failures

Problem: Tests fail with database errors

Solutions:

  • Ensure services are running: docker-compose up -d
  • Check service health: docker-compose ps
  • Tables are auto-created per test, schema is shared with dev
  • Clear all data (dev and test): docker-compose down -v (warning: destroys dev data too)

Problem: Integration tests timeout

Solutions:

  • Increase timeout in pytest config
  • Check service health: docker-compose ps
  • Review test logs for hanging operations

Import Errors

Problem: ModuleNotFoundError: No module named 'secular'

Solutions:

  • Install dependencies: uv sync
  • Activate virtual environment: source venv/bin/activate
  • Ensure you're in project root directory

Performance Issues

Problem: Slow agent execution

Solutions:

  • Enable debug logging to identify bottlenecks
  • Check LLM response times in usage metrics
  • Profile memory operations (Redis/Postgres)
  • Consider using parallel tool execution
  • Review context window size and pruning settings

Problem: High memory usage

Solutions:

  • Reduce context.max_messages in settings
  • Enable context summarization
  • Check for memory leaks in long-running agents
  • Review Redis cache TTL settings

Contributing

Contributions are welcome! Please ensure:

  • All tests pass: uv run pytest
  • Code follows style guide: uv run ruff format && uv run ruff check
  • Type checking passes: uv run mypy secular
  • Pre-commit hooks pass: uv run pre-commit run --all-files
  • New features include tests (both unit and integration)
  • Documentation is updated for new features
  • Files remain under 500 lines (modularize if needed)

License

MIT License - see LICENSE for details.

Support

For issues and questions, please open an issue on GitHub.

Releases

No releases published

Packages

 
 
 

Contributors

Languages