Chat Interface

Chat Interface

Relevant source files

The following files were used as context for generating this wiki page:

• .gitignore
• cmd/axis/agent.go
• cmd/axis/chat.go
• cmd/axis/chat_test.go
• cmd/axis/command_surface_test.go
• internal/chat/client.go
• internal/chat/engine.go
• internal/chat/engine_test.go
• internal/chat/models.go
• internal/chat/models_test.go
• internal/chat/ollama.go
• internal/chat/wave1_test.go

The Chat Interface provides a cluster-aware Read-Eval-Print Loop (REPL) and single-shot command interface for interacting with Large Language Models (LLMs). It integrates the AXIS fact plane, placement engine, and transport layers to provide a seamless advisory experience, including intelligent auto-routing of inference to remote nodes via SSH tunnels.

Architecture and Data Flow

The chat interface acts as a consumer of the AXIS Advisory Plane (Layer 5). It utilizes the internal/chat package to manage conversations and communicate with Ollama-compatible backends.

Inference Routing and Tunnelling

A key feature of axis chat is its ability to route inference requests to the most suitable node in the cluster. If the local machine lacks the resources for a specific model, AXIS identifies a remote node, establishes an SSH tunnel, and forwards the local Ollama port (11434) to that node.

Chat Execution Pipeline

The following diagram illustrates the flow from a user query to the generation of a response, highlighting the integration with the placement and transport systems.

Figure 1: Chat Execution and Auto-Routing Flow

sequenceDiagram
    participant User
    participant CLI as "cmd/axis/chat.go"
    participant Placement as "internal/placement"
    participant Transport as "internal/transport/SSHExecutor"
    participant Client as "internal/chat/Client"
    participant Ollama as "Remote Ollama Daemon"

    User->>CLI: axis chat "How is the cluster?"
    CLI->>Placement: InferRequirements("ollama run <model>")
    CLI->>Placement: SelectBestNode(reqs, snapshot, state)
    
    alt Node is Remote
        CLI->>Transport: NewSSHExecutor(target)
        CLI->>Transport: ForwardLocal(ctx, 0, 11434)
        Transport-->>CLI: boundPort (e.g. 56789)
        Note over CLI, Client: Endpoint updated to localhost:56789
    end

    CLI->>Client: ChatStream(ctx, msgs, tools, stdout)
    Client->>Ollama: POST /api/chat (via Tunnel)
    Ollama-->>Client: NDJSON Stream
    Client-->>User: Streaming Response

Loading

Sources: cmd/axis/chat.go:80-101, internal/chat/client.go:71-144, internal/transport/ssh.go:1-50

---

Key Components

1. The REPL and Command Handling

The interactive session is managed using the readline library, providing history and tab completion.

• Slash Commands: Intercepted by handleSlashCommand to perform administrative tasks without LLM intervention.
  • /status: Injects a chat.BuildClusterSummary into the view.
  • /model <tag>: Switches the active model by creating a new chat.NewClient.
  • /clear: Resets the chat.Conversation while preserving the system prompt.

2. Model Resolution Hierarchy

AXIS follows a deterministic hierarchy to select the chat model:

1. Flag: User-provided --model argument.
2. Config: chat.default_model in ~/.axis/nodes.yaml.
3. Auto-detection: chat.ResolveDefaultModel queries the local Ollama instance and matches against recommendedLocalModels (e.g., qwen3.5:4b, llama3.1:8b).

3. Conversation Management

The chat.Conversation class handles history persistence and context window management.

• Persistence: Conversations are saved to ~/.axis/history/chat.json via SaveToFile and can be reloaded using the --resume flag.
• Context Compaction: To prevent exceeding context limits, internal/chat/message.go implements CompactsOldToolResults, which truncates large tool outputs once they are no longer in the immediate message history.

Figure 2: Chat Logic Entity Map

classDiagram
    class ChatCmd {
        <<file: cmd/axis/chat.go>>
        +runPlainREPL()
        +handleSlashCommand()
    }
    class Client {
        <<file: internal/chat/client.go>>
        +Endpoint string
        +Model string
        +ChatStream()
        +EnsureRunning()
    }
    class Conversation {
        <<file: internal/chat/message.go>>
        +Messages []Message
        +Append(Message)
        +SaveToFile(path)
        +EstimateTokens()
    }
    class ModelCatalog {
        <<file: internal/chat/models.go>>
        +RecommendedLocal []ModelOption
        +Installed []string
    }

    ChatCmd --> Client : initializes
    ChatCmd --> Conversation : manages
    ChatCmd ..> ModelCatalog : uses for /models
    Client ..> Conversation : consumes messages

Loading

Sources: cmd/axis/chat.go:160-200, internal/chat/client.go:31-44, internal/chat/message.go:1-50, internal/chat/models.go:21-44

---

Technical Implementation Details

Intelligent Endpoint Management

When auto-routing is active, AXIS utilizes executor.ForwardLocal to create a secure bridge to a remote inference provider. This allows the local chat.Client to remain agnostic of the physical location of the LLM, simply targeting a dynamically assigned local port.

Function	File	Purpose
resolveChatModel	cmd/axis/chat.go:318-335	Implements the resolution hierarchy (Flag > Config > Auto-detect).
BuildSystemPrompt	internal/chat/system.go:20-50	Constructs the "Absolute Commander" prompt with optional cluster facts.
EnsureRunning	internal/chat/client.go:148-230	Automatically calls ollama serve if the daemon is unresponsive.
ChatStream	internal/chat/client.go:71-144	Handles NDJSON streaming from the Ollama /api/chat endpoint.

Model Recommendations

The system maintains a list of recommendedLocalModels in internal/chat/models.go. These are prioritized based on their ability to support tool-calling (via toolCapablePrefixes) and their parameter size relative to typical AXIS node capabilities.

Model Family	Role	Capability
qwen3.5:4b	Default	Tool-use, Balanced
llama3.1:8b	Alternative	Strong Reasoning
qwen3:0.6b	Fallback	Ultra-lightweight

Sources: internal/chat/models.go:30-44, internal/chat/models.go:178-185

---

Summary of Operation

1. Initialization: The CLI loads the runtimectx to get the current ClusterSnapshot.
2. Routing: The placement.SelectBestNode function determines if the selected model should run locally or via an SSH tunnel to a more powerful node.
3. Context Injection: If --use-context is passed, chat.BuildClusterSummary generates a text representation of the cluster (Node count, Free RAM, Reachable nodes) to be injected into the system prompt.
4. REPL: The user enters queries; the system streams responses while maintaining a persistent JSON history for multi-turn coherence.

Sources: cmd/axis/chat.go:60-120, internal/chat/system.go:52-80

---