Skip to content

Architecture 3 Providers

Jump to bottom
Danny Volz edited this page Jul 9, 2025 · 3 revisions

Providers Module Architecture

Code References

Overview

The Providers module abstracts different AI model providers behind a common interface, allowing Coda Code Assistant to work with multiple LLM backends. The module implements a provider registry system with factory pattern for consistent provider creation and management. The base interface is defined in coda/providers/base.py.

Module Structure

coda/providers/
├── __init__.py           # Module exports (verified)
├── base.py              # Abstract base provider and data models (verified)
├── registry.py          # Provider registration and factory (verified)
├── litellm_provider.py  # LiteLLM integration (100+ providers) (verified)
├── ollama_provider.py   # Ollama local models (verified)
├── oci_genai.py        # Oracle Cloud Infrastructure GenAI (verified)
├── mock_provider.py     # Mock provider for testing (verified)
└── utils.py            # Shared utilities (verified)

Key Components

Component 1: Base Provider Interface

Location: coda/providers/base.py

Purpose: Defines the abstract interface all providers must implement

Core Methods:

  • name (property): Provider identifier
  • chat(): Synchronous chat completion
  • chat_stream(): Streaming chat completion
  • achat(): Async chat completion
  • achat_stream(): Async streaming completion
  • list_models(): Available model discovery

Component 2: Provider Registry

Location: coda/providers/registry.py

Purpose: Manages provider registration and instance caching

Key Features:

  • Static class-level registry
  • Singleton instance caching
  • Config-based cache key generation

Component 3: Provider Factory

Location: coda/providers/registry.py

Purpose: Creates provider instances with merged configuration

Key Methods:

  • __init__(): Initialize with global config
  • create(): Create provider with config merging

Component 4: Data Models

Location: coda/providers/base.py

Purpose: Common data structures for provider communication

Key Classes:

  • Message: Chat message with role and content
  • Tool: Tool definition
  • ToolCall: Tool invocation request
  • ChatCompletion: Non-streaming response
  • ChatCompletionChunk: Streaming response chunk

Implementation Details

Design Patterns Used

Abstract Factory Pattern

Implementation: coda/providers/base.py and coda/providers/registry.py

class BaseProvider(ABC):
    """Abstract base class for all providers."""
    
    @abstractmethod
    async def achat(self, messages: List[Message], ...) -> ChatCompletion:
        """Abstract method all providers must implement."""

Registry Pattern

Implementation: coda/providers/registry.py

class ProviderRegistry:
    """Registry for managing provider classes and instances."""
    
    _providers: Dict[str, Type[BaseProvider]] = {}
    _instances: Dict[str, BaseProvider] = {}
    
    @classmethod
    def register(cls, name: str, provider_class: Type[BaseProvider]) -> None:
        """Register a provider class."""
        cls._providers[name] = provider_class

Singleton Pattern with Caching

Implementation: coda/providers/registry.py

# Generate cache key from config
config_key = f"{name}_{cls._hash_config(kwargs)}"
if config_key in cls._instances:
    return cls._instances[config_key]

Provider Discovery Flow

sequenceDiagram
    participant CLI
    participant ProviderFactory
    participant ProviderRegistry
    participant Provider
    participant LLM Service
    
    CLI->>ProviderFactory: create(provider_name)
    ProviderFactory->>ProviderFactory: Merge configs
    ProviderFactory->>ProviderRegistry: get_provider(name, config)
    
    alt Provider cached
        ProviderRegistry->>CLI: Return cached instance
    else New provider
        ProviderRegistry->>Provider: Create instance
        Provider->>Provider: Initialize client
        Provider->>LLM Service: Test connection
        ProviderRegistry->>ProviderRegistry: Cache instance
        ProviderRegistry->>CLI: Return new instance
    end
    
    CLI->>Provider: list_models()
    Provider->>LLM Service: Fetch models
    LLM Service->>Provider: Model list
    Provider->>CLI: Return filtered models
Loading

Provider Implementations

LiteLLM Provider

Location: coda/providers/litellm_provider.py

Features:

  • Gateway to 100+ LLM providers
  • Unified interface for OpenAI, Anthropic, Google, etc.
  • Full async and streaming support
  • Tool/function calling support

Ollama Provider

Location: coda/providers/ollama_provider.py

Features:

  • Local model execution
  • HTTP client using httpx
  • Model management commands (pull, delete)
  • Custom streaming parser

OCI GenAI Provider

Location: coda/providers/oci_genai.py

Features:

  • Oracle Cloud integration
  • Model discovery with caching
  • Provider-specific formatting (Cohere vs Generic)
  • Tool calling for Cohere models

Mock Provider

Location: coda/providers/mock_provider.py

Features:

  • Testing and development
  • Context-aware responses
  • Simulated streaming
  • Configurable behaviors

Configuration

Config File: ~/.config/coda/config.toml Config Class: coda/configuration.py:CodaConfig

Configuration Hierarchy

  1. Method parameters (highest priority)
  2. Provider-specific config section
  3. Factory configuration
  4. Global configuration
  5. Environment variables (lowest priority)

Provider-Specific Configuration

# From config.example.toml
[providers.oci_genai]
compartment_id = "ocid1.compartment.oc1.."
model = "cohere.command-r-plus"

[providers.ollama]
base_url = "http://localhost:11434"
model = "llama3"

[providers.litellm]
model = "gpt-4"
api_key = "${OPENAI_API_KEY}"

Dependencies

Internal Dependencies

  • coda.configuration: Config management (used in all providers)
  • coda.constants: Provider name constants

External Dependencies

  • httpx>=0.24.0: HTTP client for Ollama
  • litellm>=1.0.0: Multi-provider gateway
  • oci>=2.88.0: Oracle Cloud SDK
  • pydantic>=2.0: Data validation

Testing

Test Coverage

  • Unit tests: tests/unit/test_providers.py
  • Provider tests: tests/providers/test_*.py
  • Integration tests: tests/integration/test_*_integration.py

Key Test Cases

  1. Registry Tests: Provider registration and caching
  2. Mock Provider: Comprehensive behavior testing
  3. Streaming Tests: Streaming response handling

Error Handling

Common Exceptions

class ProviderError(Exception):
    """Base exception for provider errors."""
    
class ModelNotFoundError(ProviderError):
    """Raised when requested model is not available."""
    
class AuthenticationError(ProviderError):
    """Raised when authentication fails."""

Provider-Specific Error Handling

except httpx.HTTPStatusError as e:
    if e.response.status_code == 404:
        raise ModelNotFoundError(
            f"Model '{model}' not found. "
            f"Run 'ollama pull {model}' first."
        )

Performance Considerations

  1. Instance Caching: Providers cached by config hash
  2. Model List Caching: 24-hour cache for OCI models
  3. Connection Pooling: HTTP clients reused
  4. Async First: All providers support async operations

Security Considerations

  1. Credential Management: API keys from environment variables
  2. OCI Authentication: Uses OCI config file (~/.oci/config)
  3. No Credential Storage: Keys never persisted to disk
  4. Secure Defaults: HTTPS preferred for all connections

Examples

Basic Usage

from coda.providers import ProviderFactory

factory = ProviderFactory(config)
provider = factory.create("ollama", model="llama3")

response = await provider.achat([
    {"role": "user", "content": "Hello!"}
])

Streaming Usage

async for chunk in provider.achat_stream(messages):
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Integration Points

With Other Modules

  1. CLI Module: Provider initialization in provider_manager.py
  2. Agent Module: Providers used for AI completions
  3. Session Module: Provider responses stored in message history

Extension Points

  1. New Providers: Extend BaseProvider and register
  2. Custom Models: Override list_models() for filtering
  3. Response Processing: Override chat methods for custom logic

Known Limitations

  1. Tool Calling: Only supported by some providers (see provider docs)
  2. Model Names: Must match provider's expected format
  3. Streaming Variability: Chunk sizes vary by provider

References

Related Documentation

Source Files

All files referenced in this document:

Clone this wiki locally