From d4144b257fb1f0ca7d7f9ea48ad44e2f41bf8e7c Mon Sep 17 00:00:00 2001 From: 0xbbjoker <0xbbjoker@proton.me> Date: Tue, 15 Jul 2025 21:46:16 +0200 Subject: [PATCH] remove local-ai from docs --- docs.json | 3 +- plugins/llm.mdx | 157 ++++++++++++++++++++++----------- plugins/llm/local-ai.mdx | 182 --------------------------------------- plugins/llm/ollama.mdx | 3 + plugins/overview.mdx | 4 +- 5 files changed, 113 insertions(+), 236 deletions(-) delete mode 100644 plugins/llm/local-ai.mdx diff --git a/docs.json b/docs.json index 8660382..da2f59a 100644 --- a/docs.json +++ b/docs.json @@ -259,8 +259,7 @@ "plugins/llm/anthropic", "plugins/llm/google-genai", "plugins/llm/ollama", - "plugins/llm/openrouter", - "plugins/llm/local-ai" + "plugins/llm/openrouter" ] }, { diff --git a/plugins/llm.mdx b/plugins/llm.mdx index 2794867..287e16d 100644 --- a/plugins/llm.mdx +++ b/plugins/llm.mdx @@ -10,24 +10,33 @@ ElizaOS uses a plugin-based architecture for integrating different Language Mode ### Model Types -ElizaOS supports three types of model operations: +ElizaOS supports many types of AI operations. Here are the most common ones: -1. **TEXT_GENERATION** - Generating conversational responses -2. **EMBEDDING** - Creating vector embeddings for memory and similarity search -3. **OBJECT_GENERATION** - Structured output generation (JSON/XML) +1. **TEXT_GENERATION** (`TEXT_SMALL`, `TEXT_LARGE`) - Having conversations and generating responses +2. **TEXT_EMBEDDING** - Converting text into numbers for memory and search +3. **OBJECT_GENERATION** (`OBJECT_SMALL`, `OBJECT_LARGE`) - Creating structured data like JSON + +Think of it like different tools in a toolbox: +- **Text Generation** = Having a conversation +- **Embeddings** = Creating a "fingerprint" of text for finding similar things later +- **Object Generation** = Filling out forms with specific information ### Plugin Capabilities -Not all LLM plugins support all model types: +Not all LLM plugins support all model types. Here's what each can do: + +| Plugin | Text Chat | Embeddings | Structured Output | Runs Offline | +|--------|-----------|------------|-------------------|--------------| +| OpenAI | ✅ | ✅ | ✅ | ❌ | +| Anthropic | ✅ | ❌ | ✅ | ❌ | +| Google GenAI | ✅ | ✅ | ✅ | ❌ | +| Ollama | ✅ | ✅ | ✅ | ✅ | +| OpenRouter | ✅ | ❌ | ✅ | ❌ | -| Plugin | Text Generation | Embeddings | Object Generation | -|--------|----------------|------------|-------------------| -| OpenAI | ✅ | ✅ | ✅ | -| Anthropic | ✅ | ❌ | ✅ | -| Google GenAI | ✅ | ✅ | ✅ | -| Ollama | ✅ | ✅ | ✅ | -| OpenRouter | ✅ | ❌ | ✅ | -| Local AI | ✅ | ✅ | ✅ | +**Key Points:** +- 🌟 **OpenAI & Google GenAI** = Do everything (jack of all trades) +- 💬 **Anthropic & OpenRouter** = Amazing at chat, need help with embeddings +- 🏠 **Ollama** = Your local hero - does almost everything, no internet needed! ## Plugin Loading Order @@ -39,57 +48,109 @@ plugins: [ '@elizaos/plugin-sql', // Text-only plugins (no embedding support) - ...(process.env.ANTHROPIC_API_KEY ? ['@elizaos/plugin-anthropic'] : []), - ...(process.env.OPENROUTER_API_KEY ? ['@elizaos/plugin-openrouter'] : []), - - // Embedding-capable plugins last (lowest priority for embedding fallback) - ...(process.env.OPENAI_API_KEY ? ['@elizaos/plugin-openai'] : []), - ...(process.env.OLLAMA_API_ENDPOINT ? ['@elizaos/plugin-ollama'] : []), - ...(process.env.GOOGLE_GENERATIVE_AI_API_KEY ? ['@elizaos/plugin-google-genai'] : []), - - // Fallback when no other LLM is configured - ...(!process.env.GOOGLE_GENERATIVE_AI_API_KEY && - !process.env.OLLAMA_API_ENDPOINT && - !process.env.OPENAI_API_KEY - ? ['@elizaos/plugin-local-ai'] - : []), + ...(process.env.ANTHROPIC_API_KEY?.trim() ? ['@elizaos/plugin-anthropic'] : []), + ...(process.env.OPENROUTER_API_KEY?.trim() ? ['@elizaos/plugin-openrouter'] : []), + + // Embedding-capable plugins (optional, based on available credentials) + ...(process.env.OPENAI_API_KEY?.trim() ? ['@elizaos/plugin-openai'] : []), + ...(process.env.GOOGLE_GENERATIVE_AI_API_KEY?.trim() ? ['@elizaos/plugin-google-genai'] : []), + + // Ollama as fallback (only if no main LLM providers are configured) + ...(process.env.OLLAMA_API_ENDPOINT?.trim() ? ['@elizaos/plugin-ollama'] : []), ] ``` ### Understanding the Order -1. **Text-only plugins first** - Anthropic and OpenRouter are loaded first for text generation -2. **Embedding-capable plugins last** - These serve as fallbacks for embedding operations -3. **Local AI as ultimate fallback** - Only loads when no cloud providers are configured +Think of it like choosing team players - you pick specialists first, then all-rounders: + +1. **Anthropic & OpenRouter go first** - They're specialists! They're great at text generation but can't do embeddings. By loading them first, they get priority for text tasks. + +2. **OpenAI & Google GenAI come next** - These are the all-rounders! They can do everything: text generation, embeddings, and structured output. They act as fallbacks for what the specialists can't do. + +3. **Ollama comes last** - This is your local backup player! It supports almost everything (text, embeddings, objects) and runs on your computer. Perfect when cloud services aren't available. + +### Why This Order Matters + +When you ask ElizaOS to do something, it looks for the best model in order: + +- **Generate text?** → Anthropic gets first shot (if loaded) +- **Create embeddings?** → Anthropic can't, so OpenAI steps in +- **No cloud API keys?** → Ollama handles everything locally + +This smart ordering means: +- You get the best specialized models for each task +- You always have fallbacks for missing capabilities +- You can run fully offline with Ollama if needed + +### Real Example: How It Works + +Let's say you have Anthropic + OpenAI configured: + +``` +Task: "Generate a response" +1. Anthropic: "I got this!" ✅ (Priority 100 for text) +2. OpenAI: "I'm here if needed" (Priority 50) + +Task: "Create embeddings for memory" +1. Anthropic: "Sorry, can't do that" ❌ +2. OpenAI: "No problem, I'll handle it!" ✅ + +Task: "Generate structured JSON" +1. Anthropic: "I can do this!" ✅ (Priority 100 for objects) +2. OpenAI: "Standing by" (Priority 50) +``` ## Model Registration -Each LLM plugin registers its models with the runtime during initialization: +When plugins load, they "register" what they can do. It's like signing up for different jobs: ```typescript -// Example from a plugin's init method -runtime.registerModel({ - type: ModelType.TEXT_GENERATION, - handler: generateText, - provider: 'openai', - priority: 1 -}); +// Each plugin says "I can do this!" +runtime.registerModel( + ModelType.TEXT_LARGE, // What type of work + generateText, // How to do it + 'anthropic', // Who's doing it + 100 // Priority (higher = goes first) +); ``` -### Priority System +### How ElizaOS Picks the Right Model + +When you ask ElizaOS to do something, it: -Models are selected based on: -1. **Explicit provider** - If specified, uses that provider's model -2. **Priority** - Higher priority models are preferred -3. **Registration order** - First registered wins for same priority +1. **Checks what type of work it is** (text? embeddings? objects?) +2. **Looks at who signed up** for that job +3. **Picks based on priority** (higher number goes first) +4. **If tied, first registered wins** + +**Example**: You ask for text generation +- Anthropic registered with priority 100 ✅ (wins!) +- OpenAI registered with priority 50 +- Ollama registered with priority 10 + +But for embeddings: +- Anthropic didn't register ❌ (can't do it) +- OpenAI registered with priority 50 ✅ (wins!) +- Ollama registered with priority 10 ## Embedding Fallback Strategy -Since not all plugins support embeddings, ElizaOS uses a fallback strategy: +Remember: Not all plugins can create embeddings! Here's how ElizaOS handles this: + +**The Problem**: +- You're using Anthropic (great at chat, can't do embeddings) +- But ElizaOS needs embeddings for memory and search + +**The Solution**: +ElizaOS automatically finds another plugin that CAN do embeddings! ```typescript -// If primary plugin (e.g., Anthropic) doesn't support embeddings, -// the runtime will automatically use the next available embedding provider +// What happens behind the scenes: +// 1. "I need embeddings!" +// 2. "Can Anthropic do it?" → No ❌ +// 3. "Can OpenAI do it?" → Yes ✅ +// 4. "OpenAI, you're up!" ``` ### Common Patterns @@ -149,9 +210,6 @@ OPENROUTER_API_KEY=sk-or-... OPENROUTER_SMALL_MODEL=google/gemini-2.0-flash-001 # Optional: any available model OPENROUTER_LARGE_MODEL=anthropic/claude-3-opus # Optional: any available model -# Local AI (no API key needed) -``` - **Important**: The model names shown are examples. You can use any model available from each provider. ### Character-Specific Secrets @@ -182,7 +240,6 @@ You can also configure API keys per character: ### Local/Self-Hosted - [Ollama Plugin](./ollama) - Run models locally with Ollama -- [Local AI Plugin](./local-ai) - Fully offline operation ## Best Practices diff --git a/plugins/llm/local-ai.mdx b/plugins/llm/local-ai.mdx deleted file mode 100644 index b0a99e5..0000000 --- a/plugins/llm/local-ai.mdx +++ /dev/null @@ -1,182 +0,0 @@ ---- -title: "Local AI Plugin" -description: "Fully offline AI capabilities for ElizaOS" ---- - -## Features - -- **100% offline** - No internet connection required -- **Privacy-first** - Data never leaves your machine -- **No API keys** - Zero configuration needed -- **Multimodal** - Text, embeddings, vision, and speech - -## Installation - -```bash -elizaos plugins add @elizaos/plugin-local-ai -``` - -## Automatic Activation - -Local AI serves as the ultimate fallback when no cloud providers are configured: - -```typescript -// Automatically loads when these are not set: -// - GOOGLE_GENERATIVE_AI_API_KEY -// - OLLAMA_API_ENDPOINT -// - OPENAI_API_KEY -``` - -## Configuration - -### Environment Variables - -```bash -# Optional configuration -LOCAL_AI_MODEL_PATH=/path/to/models -LOCAL_AI_THREADS=4 -LOCAL_AI_CONTEXT_SIZE=4096 -``` - -### Character Configuration - -```json -{ - "name": "MyAgent", - "plugins": ["@elizaos/plugin-local-ai"] -} -``` - -## Supported Operations - -| Operation | Technology | Notes | -|-----------|------------|-------| -| TEXT_GENERATION | llama.cpp | Various model sizes | -| EMBEDDING | Local transformers | Sentence embeddings | -| VISION | Local vision models | Image description | -| SPEECH | Whisper + TTS | Transcription & synthesis | - -## Model Management - -The plugin automatically downloads required models on first use: - -```typescript -// Models are cached in: -// ~/.eliza/models/ -``` - -### Available Models - -#### Text Generation -- Small: 1-3B parameter models -- Medium: 7B parameter models -- Large: 13B+ parameter models - -#### Embeddings -- Sentence transformers -- MiniLM variants - -#### Vision -- BLIP for image captioning -- CLIP for image understanding - -## Performance Optimization - -### CPU Optimization -```bash -# Use more threads -LOCAL_AI_THREADS=8 - -# Enable AVX2 (if supported) -LOCAL_AI_USE_AVX2=true -``` - -### Memory Management -```bash -# Limit context size -LOCAL_AI_CONTEXT_SIZE=2048 - -# Use quantized models -LOCAL_AI_QUANTIZATION=q4_0 -``` - -## Hardware Requirements - -| Feature | Minimum RAM | Recommended | -|---------|-------------|-------------| -| Text (Small) | 4GB | 8GB | -| Text (Medium) | 8GB | 16GB | -| Embeddings | 2GB | 4GB | -| Vision | 4GB | 8GB | -| All Features | 16GB | 32GB | - -## Common Use Cases - -### 1. Development Environment -```json -{ - "plugins": ["@elizaos/plugin-local-ai"], - "settings": { - "local_ai": { - "model_size": "small", - "fast_mode": true - } - } -} -``` - -### 2. Privacy-Critical Applications -```json -{ - "plugins": ["@elizaos/plugin-local-ai"], - "settings": { - "local_ai": { - "model_size": "large", - "disable_telemetry": true - } - } -} -``` - -### 3. Offline Deployment -```json -{ - "plugins": ["@elizaos/plugin-local-ai"], - "settings": { - "local_ai": { - "preload_models": true, - "cache_responses": true - } - } -} -``` - -## Limitations - -- Slower than cloud APIs -- Limited model selection -- Higher memory usage -- CPU-bound performance - -## Troubleshooting - -### Model Download Issues -```bash -# Clear model cache -rm -rf ~/.eliza/models/ - -# Download models manually -eliza download-models -``` - -### Performance Issues -1. Use smaller models -2. Enable quantization -3. Reduce context size -4. Add more RAM/CPU - -## External Resources - -- [Plugin Source](https://github.com/elizaos/eliza/tree/main/packages/plugin-local-ai) -- [Model Compatibility List](https://github.com/elizaos/eliza/wiki/local-ai-models) -- [Performance Tuning Guide](https://github.com/elizaos/eliza/wiki/local-ai-performance) \ No newline at end of file diff --git a/plugins/llm/ollama.mdx b/plugins/llm/ollama.mdx index 9b7ec17..f5c0427 100644 --- a/plugins/llm/ollama.mdx +++ b/plugins/llm/ollama.mdx @@ -3,12 +3,15 @@ title: "Ollama Plugin" description: "Local model execution via Ollama for ElizaOS" --- +The Ollama plugin provides local model execution and can serve as a fallback option when cloud-based LLM providers are not configured. It requires running an Ollama server locally. + ## Features - **Local execution** - No API keys or internet required - **Multiple models** - Support for Llama, Mistral, Gemma, and more - **Full model types** - Text, embeddings, and objects - **Cost-free** - No API charges +- **Fallback option** - Can serve as a local fallback when cloud providers are unavailable ## Prerequisites diff --git a/plugins/overview.mdx b/plugins/overview.mdx index de06b7b..f4c5f69 100644 --- a/plugins/overview.mdx +++ b/plugins/overview.mdx @@ -71,8 +71,8 @@ Choose from various language model providers: Claude 3 and other Anthropic models. - - Self-hosted models for privacy and control. + + OpenRouter models for advanced routing and customization.