Agent observability framework for elizaOS — automatically tracks provider output and enables enhanced reporting for cognitive plugins.
Observatory answers the question: "What's influencing my agent?"
When building AI agents, understanding why they respond the way they do is crucial. Observatory automatically tracks all provider output and enables plugins to enhance their entries with educational content, metrics, and explanations.
┌─────────────────────────────────────────────────────────────┐
│ runtime.composeState() │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────┐ │
│ │Provider A│ │Provider B│ │Provider C│ │ COLLECTOR │ │
│ │ │ │(Piaget) │ │ │ │(Observatory)│ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ └─────┬──────┘ │
│ │ │ │ │ │
│ └─────────────┴─────────────┴──────────────┘ │
│ ↓ │
│ state.data.providers (cached) │
│ ↓ │
│ COLLECTOR reads all outputs │
└──────────────────────────┬──────────────────────────────────┘
↓
┌───────────────────────┐
│ ObservatoryService │
│ ┌─────────────────┐ │
│ │ Entry Store │ │
│ │ (max 1000) │ │
│ └─────────────────┘ │
│ ┌─────────────────┐ │
│ │Enhancer Registry│ │
│ │ Piaget → ✨ │ │
│ └─────────────────┘ │
└───────────────────────┘
bun add @elizaos/plugin-observatoryimport { observatoryPlugin } from '@elizaos/plugin-observatory';
const agent = new AgentRuntime({
plugins: [
observatoryPlugin,
// ... other plugins
],
});Users can query Observatory through conversation:
- "What's influencing you?"
- "Show me your providers"
- "How are you being steered?"
const observatory = runtime.getService('observatory') as ObservatoryService;
// Get all entries
const entries = observatory.getEntries(10);
// Get entries for a specific provider
const cognitive = observatory.getEntriesByProvider('COGNITIVE_STATE');
// Get only enhanced entries
const enhanced = observatory.getEnhancedEntries();
// Get summary statistics
const summary = observatory.getSummary();Make your provider's output meaningful by registering an enhancer:
// In your plugin's init()
const observatory = runtime.getService('observatory');
if (observatory) {
observatory.registerEnhancer({
providerName: 'YOUR_PROVIDER_NAME', // Must match provider.name
enhance: (entry, runtime) => ({
// Required: Human-readable summary
description: 'What your provider does this interaction',
// Required: What aspects your provider influences
influences: [
{ aspect: 'tone', description: 'Cheerful greeting', strength: 0.7 },
{ aspect: 'knowledge', description: 'User preferences loaded', strength: 0.8 },
],
// Optional: Educational content about your framework
education: {
framework: 'Your Theoretical Framework',
concepts: [
{
name: 'Key Concept',
explanation: 'What this concept means in practice',
quote: 'Optional quote from framework source',
},
],
},
// Optional: Metrics for quantitative tracking
metrics: [
{ name: 'metric_name', value: 0.5, explanation: 'What this measures' },
],
// Optional: The "delta" - what's different because of your plugin
counterfactual: 'Without this plugin, responses would be...',
}),
});
}tone- How the agent sounds (emotion, formality)knowledge- What the agent knows (facts, beliefs)behavior- How the agent acts (actions, responses)personality- Who the agent is (character, style)memory- What the agent remembers (history, context)reasoning- How the agent thinks (logic, approach)
The plugin-piaget provides a rich enhancer showing how cognitive psychology concepts influence the agent:
// Piaget enhancer output example
{
description: "Cognitive layer: exploratory mode, curious emotion, 45% disequilibrium",
influences: [
{ aspect: 'tone', description: 'Curious emotional coloring', strength: 0.7 },
{ aspect: 'behavior', description: 'Exploratory engagement mode', strength: 0.8 },
{ aspect: 'reasoning', description: 'Low disequilibrium - confident reasoning', strength: 0.45 },
],
education: {
framework: 'Piagetian Developmental Psychology',
concepts: [
{
name: 'Play Stance',
explanation: 'The agent is in discovery mode - casting a wide net, following curiosity.',
quote: 'Each time one prematurely teaches a child something he could have discovered himself, that child is kept from inventing it.',
},
{
name: 'Disequilibrium',
explanation: 'Moderate cognitive tension (45%). The agent is curious and open to learning.',
quote: "Intelligence is what you use when you don't know what to do.",
},
],
},
metrics: [
{ name: 'Disequilibrium', value: 0.45, explanation: 'Cognitive tension (0=certain, 1=uncertain)' },
{ name: 'Schema Count', value: 12, explanation: 'Mental frameworks held' },
{ name: 'Maturity', value: 0.35, explanation: 'Developmental progress' },
],
counterfactual: 'Without this cognitive layer:\n• Responses would be more direct, less wondering\n• Would not draw on 12 learned beliefs',
}import { ObservatoryService } from '@elizaos/plugin-observatory';
// Custom configuration
const observatory = new ObservatoryService({
maxEntries: 500, // Default: 1000
verbose: true, // Default: false
});| Method | Description |
|---|---|
recordProviderOutput(entry) |
Record a provider's output (called by collector) |
registerEnhancer(enhancer) |
Register an enhancer for a provider |
unregisterEnhancer(name) |
Remove an enhancer |
hasEnhancer(name) |
Check if enhancer exists |
getEntries(limit?) |
Get recent entries (most recent first) |
getEntriesByProvider(name, limit?) |
Filter by provider |
getEnhancedEntries(limit?) |
Get only enhanced entries |
getEntriesByInteraction(id) |
Filter by interaction |
getLatestEntry(provider) |
Get most recent for provider |
getSummary() |
Get aggregate statistics |
getTotalTokens() |
Get total token count |
getProviderCount() |
Get unique provider count |
clear() |
Clear all entries |
interface ProviderEntry {
providerName: string;
timestamp: number;
interactionId: UUID;
output: {
text: string;
textLength: number;
tokensEstimate: number;
values: Record<string, unknown>;
data: Record<string, unknown>;
};
enhanced?: EnhancedData;
}
interface EnhancedData {
description: string;
influences: InfluenceClaim[];
education?: EducationalContent;
metrics?: ProviderMetric[];
counterfactual?: string;
}
interface ProviderEnhancer {
providerName: string;
enhance: (entry: ProviderEntry, runtime: IAgentRuntime) => EnhancedData;
}- Debug agent behavior without adding console.logs everywhere
- Track which providers are most influential
- Understand the cognitive layers at play
- Study how different frameworks affect agent responses
- Compare influence patterns across conversations
- Measure provider effectiveness quantitatively
- Ask "What's influencing you?" to understand agent behavior
- Learn about the theoretical frameworks behind the agent
- See concrete metrics about agent state
Observatory tracks ALL providers automatically without requiring their cooperation. This enables observability even for third-party providers that don't implement enhancers.
Not all providers need rich educational content. Basic tracking (text, tokens, values) is enough for most debugging. Enhancement adds value for cognitive/learning-focused plugins.
We use a late-running provider that reads state.data.providers rather than monkey-patching. This is less invasive and uses existing elizaOS infrastructure.
Memory is finite. We keep the most recent entries (default 1000) and drop older ones. This prevents unbounded memory growth while maintaining useful history.
MIT