diff --git a/docs/chat-agent-guide.md b/docs/chat-agent-guide.md deleted file mode 100644 index 603bb993b..000000000 --- a/docs/chat-agent-guide.md +++ /dev/null @@ -1,252 +0,0 @@ -# Chat Agent Guide - -## What is the Chat Agent? - -The Chat Agent is your AI-powered assistant that helps you manage your customer relationships, automate workflows, and access your business data through natural conversation. Simply chat with it like you would with a colleague, and it will understand your requests and take action. - -## What Can the Chat Agent Do? - -### πŸ’¬ Natural Conversation - -- **Multi-turn conversations**: The agent remembers context from your entire conversation -- **File attachments**: Upload images, PDFs, documents, and the agent will analyze them -- **Smart memory**: Automatically summarizes long conversations to maintain context -- **Real-time responses**: See the agent's thinking process as it works - -### πŸ“Š Customer & Product Management - -#### Customer Operations - -- Search for customers by email, ID, or any field -- List all customers with pagination -- View customer details, status, purchase history -- Filter customers by status, source, locale, or custom metadata - -#### Product Operations - -- Browse your product catalog -- Search products by ID, name, category -- View product details, pricing, stock levels -- Filter products by status, category, tags - -**Examples:** - -- "Show me all customers with status 'churned'" -- "Find the customer with email `john@example.com`" -- "List all active products in the 'electronics' category" -- "How many customers do I have?" - -### πŸ” Knowledge Base & Search - -#### RAG Search (Semantic Search) - -- Search your knowledge base using natural language -- Find relevant information from uploaded documents -- Get answers from your business policies, procedures, documentation -- Semantic understanding - finds meaning, not just keywords - -#### RAG Write (Knowledge Management) - -- Save new information to your knowledge base -- Update outdated information -- Correct mistakes in the knowledge base -- Store business policies, procedures, FAQs - -**Examples:** - -- "What is our return policy?" -- "Remember: Our store hours are 9am-5pm on weekdays" -- "That's wrong. The correct shipping policy is free shipping over $50" -- "Find information about sustainability in our products" - -### 🌐 Web & Research - -#### Web Search - -- Search the internet for current information -- Filter by time range (day, week, month, year) -- Search specific sites or categories -- Get news, research, technical information - -#### Web Scraping - -- Extract content from any webpage -- Read articles, documentation, blog posts -- Get structured data from websites - -**Examples:** - -- "Search for recent news about AI in customer service" -- "What's on the homepage of example.com?" -- "Find research papers about customer churn prediction" - -### πŸ“„ File Operations - -#### PDF Operations - -- **Generate PDFs**: Create PDFs from Markdown, HTML, or web pages -- **Parse PDFs**: Extract text from uploaded PDF files -- Custom formatting, margins, page sizes - -#### Image Generation - -- Create screenshots from Markdown, HTML, or URLs -- Generate images from content -- Custom dimensions and styling - -#### Excel Generation - -- Export data to Excel spreadsheets -- Multiple sheets, custom headers -- Perfect for customer lists, product catalogs, reports - -#### PowerPoint & Word - -- Generate PowerPoint presentations -- Create Word documents -- Professional formatting - -**Examples:** - -- "Generate a PDF report of all churned customers" -- "Create an Excel file with all active products" -- "Take a screenshot of example.com" -- "Extract text from this PDF" (with file upload) - -### πŸ€– Workflow Automation - -#### Workflow Management - -- Create automated workflows from natural language descriptions -- View and edit existing workflows -- Search for workflow examples -- List all available automation actions -- Test and activate workflows - -#### Available Workflow Actions - -The agent can create workflows using these operations: - -- Customer operations (create, update, query, filter) -- Product operations (create, update, query, filter) -- Email operations (send, fetch, reply) -- Conversation management -- Document operations -- RAG operations (upload, sync knowledge) -- Integration with external systems (Shopify, OneDrive, etc.) -- Web scraping and crawling -- Approval workflows -- And many more... - -**Examples:** - -- "Create a workflow that sends welcome emails to new customers" -- "Show me all my active workflows" -- "Build an automation that identifies at-risk customers and sends them offers" -- "What workflow actions are available for products?" - -### 🎨 Advanced Features - -#### Context Search - -- Search through your conversation history -- Find relevant past discussions -- Semantic search across all your chats - -#### Resource Checking - -- Verify file existence and accessibility -- Check resource availability before operations - -#### Tone of Voice - -- Maintain consistent brand voice in communications -- Apply custom tone settings to generated content - -## How to Use the Chat Agent - -### Starting a Conversation - -Simply type your message in the chat interface. The agent will: -1. Understand your request -2. Use the appropriate tools to gather information -3. Provide a clear, well-formatted response - -### Uploading Files - -You can attach files to your messages: -- **Images**: The agent will analyze and describe them -- **PDFs**: The agent will extract and read the text -- **Documents**: The agent will process and understand the content - -### Getting Better Results - -#### Be specific about what you need - -- ❌ "Show me customers" -- βœ… "Show me the 10 most recent customers with status 'active'" - -#### Ask follow-up questions - -- The agent remembers your conversation context -- Build on previous responses -- Refine your requests - -#### Use natural language - -- You don't need to use technical terms -- Describe what you want in plain English -- The agent will figure out the details - -## Tips & Best Practices - -### For Data Queries - -- **Use database tools** for counting, listing, filtering by specific fields -- **Use RAG search** for semantic/meaning-based searches -- **Be specific** about which fields you need to reduce response time - -### For Workflows - -- **Describe the goal**, not the implementation -- **Provide examples** of what should trigger the workflow -- **Test workflows** before activating them - -### For File Generation - -- **Specify the format** you want (PDF, Excel, Image, etc.) -- **Provide clear content** or data to include -- **Download files immediately** - URLs may expire - -### For Knowledge Base - -- **Save important information** as you learn it -- **Correct mistakes** when you find them -- **Use clear topics** when saving information - -## Common Use Cases - -1. **Customer Analysis**: "Show me all customers who haven't purchased in 90 days" -2. **Product Reports**: "Generate an Excel file of all products with low stock" -3. **Workflow Creation**: "Create an automation that sends product recommendations to active customers" -4. **Research**: "Search for best practices in customer retention" -5. **Document Generation**: "Create a PDF report of this month's customer activity" -6. **Knowledge Management**: "What's our refund policy?" or "Remember: We offer 30-day returns" - -## Technical Details - -- **Model**: Configurable AI model (default: GPT-4 or similar) -- **Max Steps**: Up to 20 tool calls per response (configurable) -- **Context**: Maintains full conversation history with automatic summarization -- **File Support**: Images, PDFs, Word, Excel, PowerPoint, and more -- **Real-time**: Responses stream as they're generated - -## Need Help? - -The Chat Agent is designed to be intuitive and helpful. If you're unsure about something: -- Just ask! "What can you help me with?" -- Request examples: "Show me an example of creating a workflow" -- Ask for clarification: "What information do you need from me?" - -The agent will guide you through any task and explain what it's doing along the way. - diff --git a/docs/docs.json b/docs/docs.json index 9b3df3df2..afb7d2a51 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -12,10 +12,7 @@ { "group": "Guides", "pages": [ - "chat-agent-guide", - "workflow-guide", - "permissions", - "zero-downtime-deployment" + "platform-documentation" ] } ] diff --git a/docs/permissions.md b/docs/permissions.md deleted file mode 100644 index 3346d7304..000000000 --- a/docs/permissions.md +++ /dev/null @@ -1,123 +0,0 @@ -# Platform Role-Based Access Control - -This document describes the permission system for the Platform service, helping administrators understand the access scope of each role. - -## Role Overview - -| Role | Description | Intended Users | -|------|-------------|----------------| -| **Admin** | System administrator | Organization owners, IT administrators | -| **Developer** | Technical operator | Engineers, integration developers | -| **Editor** | Content editor | Content managers, customer service leads | -| **Member** | Regular member | General employees, read-only users | -| **Disabled** | Disabled account | Former employees or suspended users | - -## Permission Matrix - -### AI Chat - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| Create new chat | βœ… | βœ… | βœ… | βœ… | -| Send messages to AI | βœ… | βœ… | βœ… | βœ… | -| View chat history | βœ… | βœ… | βœ… | βœ… | -| Delete/rename chat | βœ… | βœ… | βœ… | βœ… | - -> Each user can only see their own AI chat history. - -### Knowledge Base - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View documents | βœ… | βœ… | βœ… | βœ… | -| Upload/edit/delete documents | ❌ | βœ… | βœ… | βœ… | -| View products | βœ… | βœ… | βœ… | βœ… | -| Manage products | ❌ | βœ… | βœ… | βœ… | -| View customers | βœ… | βœ… | βœ… | βœ… | -| Manage customers | ❌ | βœ… | βœ… | βœ… | -| View vendors | βœ… | βœ… | βœ… | βœ… | -| Manage vendors | ❌ | βœ… | βœ… | βœ… | -| View websites | βœ… | βœ… | βœ… | βœ… | -| Manage websites | ❌ | βœ… | βœ… | βœ… | -| View tone of voice | βœ… | βœ… | βœ… | βœ… | -| Manage tone of voice | ❌ | βœ… | βœ… | βœ… | - -### Conversations (Customer Communication) - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View conversation list | βœ… | βœ… | βœ… | βœ… | -| View conversation details | βœ… | βœ… | βœ… | βœ… | -| Reply to customers | ❌ | βœ… | βœ… | βœ… | -| Close/reopen conversations | ❌ | βœ… | βœ… | βœ… | -| Mark as spam | ❌ | βœ… | βœ… | βœ… | - -### Approvals - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View pending approvals | βœ… | βœ… | βœ… | βœ… | -| Approve/reject | ❌ | βœ… | βœ… | βœ… | - -### Automations - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View workflow list | ❌ | ❌ | βœ… | βœ… | -| Create/edit workflows | ❌ | ❌ | βœ… | βœ… | -| View execution logs | ❌ | ❌ | βœ… | βœ… | - -### Integrations - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View integrations | ❌ | ❌ | βœ… | βœ… | -| Configure email integration | ❌ | ❌ | βœ… | βœ… | -| Configure OneDrive sync | ❌ | ❌ | βœ… | βœ… | -| Manage API integrations | ❌ | ❌ | βœ… | βœ… | - -### Organization Management - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| View organization info | ❌ | ❌ | ❌ | βœ… | -| Edit organization settings | ❌ | ❌ | ❌ | βœ… | -| View member list | ❌ | ❌ | ❌ | βœ… | -| Add/remove members | ❌ | ❌ | ❌ | βœ… | -| Change member roles | ❌ | ❌ | ❌ | βœ… | - -### Account Settings - -| Feature | Member | Editor | Developer | Admin | -|---------|:------:|:------:|:---------:|:-----:| -| Edit personal info | βœ… | βœ… | βœ… | βœ… | -| Change password | βœ… | βœ… | βœ… | βœ… | - -## Role Recommendations - -### Admin -- Responsible for overall organization management -- Manages team members and permission assignments -- Configures system integrations and automations - -### Developer -- Configures and maintains automation workflows -- Manages third-party system integrations -- Has full data access permissions - -### Editor -- Manages knowledge base content -- Handles customer conversations and approvals -- Suitable for content operations and customer service teams - -### Member -- Uses AI chat to retrieve information -- Views business data (read-only) -- Suitable for employees who need to query information but don't need to edit - -## Security Notes - -1. **Principle of least privilege**: Assign users the lowest permission role that meets their work requirements -2. **Admin redundancy**: The system requires at least 2 administrators to prevent loss of management access -3. **Self-protection**: Users cannot change their own role; another administrator must make the change -4. **Data isolation**: All data access is restricted to the user's organization scope diff --git a/docs/platform-documentation.md b/docs/platform-documentation.md new file mode 100644 index 000000000..0886fe541 --- /dev/null +++ b/docs/platform-documentation.md @@ -0,0 +1,936 @@ +--- +title: Tale Platform Documentation +description: Setup, features, roles, APIs, and operations for the Tale platform. +--- + +# Tale Platform Documentation + +Tale is an open-source, self-hosted AI platform for teams that want a full-stack AI application they can own, control, and extend. It includes an intelligent chat assistant, a semantic knowledge base, customer conversation management, visual automation workflows, and a structured API layer. + +Unlike cloud-only AI products, Tale runs entirely on your own infrastructure. Your data stays on your servers. There are no per-seat fees, no vendor lock-in, and no model restrictions beyond what your API key supports. + +## Architecture at a Glance + +Tale runs as six Docker services that communicate over an internal network: + +| Service | Technology | Role | Local Port | +| --- | --- | --- | --- | +| Platform | Bun + TanStack + Convex | Web UI, real-time backend, auth, data, workflows | 3000 (via proxy) | +| RAG | Python + FastAPI | Document indexing, vector search, answer generation | 8001 | +| Crawler | Python + Playwright + Crawl4AI | Website crawling, URL discovery, file-to-text conversion | 8002 | +| Operator | Python + Playwright + LLM | Browser automation agent for web tasks | 8004 | +| Database | PostgreSQL | Persistent storage for platform data | 5432 | +| Proxy | Caddy | TLS termination and routing | 80 / 443 | + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Caddy Proxy β”‚ +β”‚ (Routes traffic to healthy backend) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Blue Services β”‚ β”‚ Green Services β”‚ +β”‚ (Active) β”‚ β”‚ (Standby) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ platform-blue β”‚ β”‚ platform-green β”‚ +β”‚ rag-blue β”‚ β”‚ rag-green β”‚ +β”‚ crawler-blue β”‚ β”‚ crawler-green β”‚ +β”‚ operator-blue β”‚ β”‚ operator-green β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β–Ό + β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” + β”‚ Shared Services β”‚ + β”‚ (Stateful) β”‚ + β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ + β”‚ db (TimescaleDB) β”‚ + β”‚ proxy (Caddy) β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +> **Note:** All communication between services stays on the internal Docker network. Only ports 80 and 443 are exposed publicly through the Caddy proxy. The database (5432) and API services (8001, 8002, 8004) are exposed on the host for local development only. + +## Key Capabilities + +- AI chat assistant with multi-turn conversations, file attachments, agent selection, and built-in tools +- Semantic knowledge base for documents, websites, products, customers, and vendors +- Customer conversations inbox with AI-assisted replies and bulk actions +- Visual automation builder with LLM steps, conditionals, loops, and scheduling +- Custom AI agents with tailored instructions, knowledge, and tools +- Role-based access control from read-only Member to full Admin +- SSO and integrations including Microsoft Entra ID, REST APIs, OneDrive sync, and SQL connectors +- Production operations with zero-downtime deployments, Prometheus metrics, and Sentry error tracking + +## Setup Guide + +This section walks you through getting Tale running locally for the first time, and then covers deploying it to a production server. The whole process takes about 15 minutes. + +### Prerequisites + +#### Required Software + +| Software | Minimum Version | Where to Get It | +| --- | --- | --- | +| Docker Desktop | 24.0+ | https://www.docker.com/products/docker-desktop | +| Git | Any recent version | https://git-scm.com | + +#### Required API Key + +Tale uses OpenRouter as its default AI gateway, which gives you access to hundreds of models through a single API key. + +1. Go to https://openrouter.ai and create a free account. +2. Navigate to Keys in your account dashboard and generate a new API key. +3. Copy the key. You will need it in Step 3. + +> **Tip:** You can use any OpenAI-compatible provider, including a local Ollama instance, by setting `OPENAI_BASE_URL` and `OPENAI_API_KEY` in your `.env`. OpenRouter is the recommended choice for its model variety and simple pricing. + +### Quick Start (Local) + +If you are on Windows or want to skip building from source, jump to the pre-built images section instead. It is faster and has the same result. + +#### Step 1: Set Up Your Local Domain + +Tale uses `tale.local` as its default local address. Add one line to your hosts file so your browser can reach it. + +On macOS or Linux: + +```bash +sudo sh -c 'echo "127.0.0.1 tale.local" >> /etc/hosts' +``` + +On Windows, run PowerShell as Administrator: + +```powershell +Add-Content -Path "C:\Windows\System32\drivers\etc\hosts" -Value "127.0.0.1 tale.local" +``` + +#### Step 2: Clone the Repository + +```bash +git clone https://github.com/tale-project/tale.git +cd tale +``` + +#### Step 3: Set Up Your `.env` File + +Copy the example environment file: + +```bash +cp .env.example .env +``` + +Open `.env` in any text editor. The file already has placeholder values for everything. You only need to fill in these required ones: + +| Variable | Required? | How to Fill It In | +| --- | --- | --- | +| `OPENAI_API_KEY` | Yes | Paste your OpenRouter API key here | +| `BETTER_AUTH_SECRET` | Yes | Generate with: `openssl rand -base64 32` | +| `ENCRYPTION_SECRET_HEX` | Yes | Generate with: `openssl rand -hex 32` | +| `DB_PASSWORD` | Yes | Choose any password for the local database | + +> **Important:** The `.env.example` file ships with example secrets that are not safe to use. You must replace `BETTER_AUTH_SECRET` and `ENCRYPTION_SECRET_HEX` with your own generated values before starting. Using the example values is a security risk even in local development. + +#### Step 4: Start the Platform + +The first time you run this, Docker builds the service images from source. This takes 3 to 5 minutes. Subsequent starts are much faster. + +```bash +docker compose up --build +``` + +Watch the logs. When you see this message, everything is ready: + +```text +πŸŽ‰ Tale Platform is running! +``` + +> **Note:** You will see a stream of `200 OK` health check messages while services are starting. Those are normal and do not mean the UI is ready. Wait for the ready message before opening your browser. + +#### Step 5: Trust the Certificate (Recommended) + +Tale generates a self-signed TLS certificate for local development. Your browser will show a security warning the first time you visit. To get rid of it permanently, run: + +```bash +docker exec tale-proxy caddy trust +``` + +Then restart your browser. + +#### Step 6: Open the App + +Go to https://tale.local in your browser. The first time you open it, you will be taken to a sign-up page to create your admin account. + +### Using Pre-Built Images (Recommended for Windows) + +This approach skips the local build entirely. Docker pulls pre-built images from GitHub and starts them directly. You only need two files from the repository: `compose.yml` and `.env`. + +Add these two lines to your `.env` file: + +```dotenv +PULL_POLICY=always +VERSION=latest +``` + +Then start without the `--build` flag: + +```bash +docker compose up +``` + +> **Tip:** To update Tale to the latest version when using pre-built images, run `docker compose down`, then `docker compose pull`, then `docker compose up`. This fetches new images without affecting your data. + +### Daily Workflow + +#### Starting the Platform + +1. Open Docker Desktop and wait until the engine status shows green. +2. In your terminal, go to the `tale` folder and run `docker compose up`. +3. Wait for the platform ready message, then open https://tale.local. + +#### Stopping the Platform + +To stop all services while keeping your data: + +```bash +docker compose down +``` + +> **Important:** Never run `docker compose down -v`. The `-v` flag deletes all Docker volumes, which permanently erases your database, uploaded documents, crawler state, and all platform data. There is no recovery from this. + +### Production Deployment + +#### Option A: Docker Compose + +For a single-server production setup, update your `.env` with these values: + +```dotenv +HOST=yourdomain.com +SITE_URL=https://yourdomain.com +TLS_MODE=letsencrypt +TLS_EMAIL=admin@yourdomain.com +PULL_POLICY=always +VERSION=latest +``` + +Make sure ports 80 and 443 are open on your server firewall. Let's Encrypt will issue and renew TLS certificates automatically. + +Then start in detached mode: + +```bash +docker compose up -d +``` + +#### Option B: Zero-Downtime Deployment + +For production environments where downtime is not acceptable, Tale ships with a blue-green deployment script. It runs two versions of stateless services at the same time, checks that the new version is healthy, then switches traffic over. + +```bash +# Deploy a specific version with no downtime +./scripts/deploy.sh deploy v1.2.0 + +# Deploy latest version +./scripts/deploy.sh deploy latest + +# Roll back to the previous version +./scripts/deploy.sh rollback + +# Check which version is currently live +./scripts/deploy.sh status +``` + +> **Note:** Zero-downtime deployment requires at least 12 GB of RAM on the server because both versions run at the same time during the switchover. The database and proxy are shared and are not duplicated. + +### Convex Dashboard Access + +Tale includes an embedded Convex backend. The Convex Dashboard lets you inspect the database, view function logs, and manage background jobs. + +1. Run this to generate an admin key: + +```bash +docker exec tale-platform /app/generate-admin-key.sh +``` + +2. Copy the key from the output. +3. Open https://tale.local/convex-dashboard in your browser. +4. Paste the admin key when prompted. + +> **Note:** The Convex Dashboard gives direct read and write access to all data. Only share admin keys with trusted team members. + +## Platform Features + +### AI Chat + +The AI Chat is the main interface for working with Tale's AI. It is a conversation workspace where you can ask questions, request actions, and explore your data in plain language. + +#### Using the Chat + +- Access: Navigate to Chat in the left sidebar. +- To start a new conversation, click the plus icon in the top toolbar or use the keyboard shortcut. +- Each conversation is saved to your history and can be searched or renamed later. + +#### Sending Messages + +Type in the input area at the bottom of the screen. The Enter key sends your message. Use Shift+Enter for a new line within a message. The input area grows automatically as you type. + +#### File Attachments + +You can attach files to any message by clicking the paperclip icon or dragging files into the chat window. Supported file types include: + +- Images: PNG, JPEG, GIF, WebP. The agent analyzes the visual content. +- Documents: PDF, DOCX, XLSX, PPTX, TXT, Markdown. The agent reads the content. +- Code files: JS, TS, Python, and most common source file formats. + +Files are uploaded before the message is sent. A loading spinner shows for each file while it uploads. + +#### Selecting an Agent + +The agent selector is in the bottom-left corner of the input area, shown as a bot icon. Use it to choose which AI agent handles your conversation. The default is the system chat agent. Custom agents your team has built also appear here. + +#### Chat History + +Click the clock icon in the top toolbar to open the history sidebar. You can: + +- Browse all past conversations, grouped by date +- Click a conversation to open it +- Double-click a conversation title to rename it inline +- Use the three-dot menu to rename or delete a conversation +- Search across all conversations with `Ctrl+K` or `Cmd+K` on Mac + +#### What the Chat Agent Can Do + +The default chat agent can handle: + +| Tool Category | What You Can Ask | +| --- | --- | +| Customer lookup | Find customers by email, ID, status, locale, or custom metadata fields | +| Product catalog | Browse products by category, search by name or ID, check stock and pricing | +| Vendor information | Look up vendor details and contact information | +| Knowledge base search | Ask questions that are answered by your uploaded documents and crawled websites | + +#### Keyboard Shortcuts + +| Action | Windows / Linux | macOS | +| --- | --- | --- | +| New chat | `Alt + Ctrl + N` | `Option + Cmd + N` | +| Search chats | `Ctrl + K` | `Cmd + K` | +| Toggle history sidebar | `Ctrl + H` | `Cmd + H` | + +### Knowledge Base + +The Knowledge Base is where Tale stores information for the AI to use. Anything you add here becomes searchable by the chat agent using meaning-based search. It is split into five sections, all accessible from the Knowledge tab in the sidebar. + +> **Note:** Editing the Knowledge Base requires the Editor role or higher. Members can view all knowledge items but cannot create, update, or delete them. + +#### Documents + +Documents are the core of the knowledge base. You can upload files directly or sync from Microsoft OneDrive. Once indexed, the content is searchable by the AI agent. + +**Uploading Documents** + +1. Navigate to Knowledge > Documents. +2. Click the Upload button in the top-right action menu. +3. Drag files into the upload area or click to browse. You can select multiple files at once. +4. Optionally assign the documents to one or more teams. This controls which team-filtered views they appear in. +5. Click Upload. Each file is queued for background processing. A status indicator shows when indexing is done. + +Supported file types: PDF, DOCX, PPTX, XLSX, TXT, Markdown, CSV, HTML, JSON, YAML, and most code file formats. + +Maximum file size: 100 MB per file. + +**Folder Organization** + +Documents can be organized into folders. Use the breadcrumb navigation at the top of the Documents table to move between folders. You can create folders during upload or from the action menu. + +**Microsoft OneDrive Sync** + +If a Microsoft account integration is configured, a Sync from OneDrive option appears in the action menu. This imports documents directly from OneDrive without downloading them to your server first. + +#### Websites + +Website tracking tells Tale's crawler to visit and index pages from a given domain on a schedule. Once indexed, the AI agent can answer questions about that site's content. + +**Adding a Website** + +1. Navigate to Knowledge > Websites and click Add Website. +2. Enter the full URL of the website, for example `https://docs.example.com`. +3. Select a scan interval. This controls how often the crawler re-checks for updated content. +4. Click Add. The crawler fetches the homepage right away and starts finding linked pages. + +| Scan Interval | Best For | +| --- | --- | +| Every hour | Sites with frequent content changes | +| Every 6 hours (default) | Documentation sites and company wikis | +| Every day | Marketing sites and blogs | +| Every 7 days | Reference sites with infrequent updates | + +#### Products + +The Products section stores your product catalog. Each product record includes a name, description, image URL, stock level, price, currency, category, and status. + +Products can be added one at a time or imported in bulk via CSV. The CSV format has no header row, with columns in this order: + +```text +name, description, imageUrl, stock, price, currency, category, status +``` + +Valid status values: `active`, `inactive`, `draft`, `archived`. Invalid values fall back to `draft`. + +#### Customers + +The Customers section stores your customer list. Each customer has an email address, locale, status, and optional custom metadata. Imported customers are set to a status of `churned` by default. + +Import via CSV with this format: + +```text +email, locale +``` + +Valid locale values: `en`, `de`, `es`, `fr`, `it`, `nl`, `pt`, `zh`. Invalid locales fall back to `en`. + +#### Vendors + +The Vendors section stores supplier and partner records. Vendor data is searchable by the AI agent and can be referenced in automated workflows. The same CSV import that works for customers also works here. + +### Conversations + +Conversations is the customer inbox. When customers contact your team through a connected channel such as email, their messages appear here as conversation threads. Your team can read, reply, close, and manage them from this one view. + +#### Conversation Statuses + +| Status | Meaning | +| --- | --- | +| Open | Active conversation that needs a response or is in progress | +| Closed | Conversation that has been resolved and marked as done | +| Spam | Messages flagged as unsolicited or irrelevant | +| Archived | Conversations kept for reference but removed from the active inbox | + +#### Replying to a Conversation + +1. Click any conversation in the list to open it in the right panel. +2. The message composer loads at the bottom. It is a rich-text editor that supports bold, italic, lists, links, and code blocks. +3. Write your reply. You can attach files using the paperclip icon in the toolbar. +4. Use the AI Improve button, if enabled, to have the AI clean up your message before sending. +5. Click Send. The message is sent through whichever channel the customer used. + +#### Bulk Actions + +Select multiple conversations using the checkbox at the top of the list. Available bulk actions: + +- Change status: close, reopen, archive, or mark as spam +- Assign to a team +- Send a message to all selected conversation participants at once + +#### Priority Filtering + +Use the filter dropdown in the toolbar to show conversations by priority. This helps surface urgent threads without scrolling through the full inbox. + +### Approvals + +The Approvals queue holds actions that need a human to review before they run. Automations and AI agents can be set up to pause at certain steps and wait for approval instead of acting on their own. + +#### Reviewing an Approval + +1. Navigate to Approvals > Pending. +2. Click any approval to see the full context: which workflow triggered it, what action it wants to take, and what data it would use. +3. Click Approve to let it proceed, or Reject to cancel it. +4. All resolved approvals, approved and rejected, are available in the Resolved tab for reference. + +### Automations + +Automations let you define and run multi-step business processes without writing backend code. A workflow is a series of steps. Each step does one thing, and steps are connected to form a complete process. + +#### Creating a Workflow + +There are two ways to create a workflow: + +##### Option 1: AI-Assisted Creation + +1. Navigate to Automations and click New Automation. +2. Enter a name and a description of what the workflow should do. The more detail you add, the better the AI can build the initial steps. +3. Click Continue. The platform creates the workflow and opens the AI Chat panel on the right where you can refine things in conversation. + +##### Option 2: Manual Visual Editor + +1. Create a new workflow as above but leave the description blank. +2. Use the Add Step button on the workflow canvas to add steps one at a time. +3. Configure each step using the side panel that appears when you click on a step. +4. Connect steps by clicking the connector handles and drawing lines between them. + +#### Step Types + +| Step Type | Color | What It Does | +| --- | --- | --- | +| Trigger | Teal | The entry point of the workflow. Defines when it starts, such as schedule, event, webhook, or manual run | +| Action | Blue | Runs an operation such as create a record, send a message, call an API, or update data | +| LLM | Purple | Sends a prompt to an AI model and passes the response to the next step | +| Condition | Orange | Checks a condition and routes execution down different branches | +| Loop | Green | Repeats a set of steps for each item in a list | + +#### Triggers + +Every workflow needs at least one trigger to know when to run. + +##### Schedule Triggers + +Run the workflow on a time schedule. You can enter a cron expression directly or use the AI assistant to generate one from plain English, such as β€œevery weekday at 9am”. + +All schedules run in UTC. Quick presets are available for every 5 minutes, hourly, daily, weekly, and monthly. + +##### Event Triggers + +Run the workflow when something happens in the platform, for example when a new customer is added, a conversation is opened, or a product's stock hits zero. Each event type can have optional filter conditions. + +##### Webhook Triggers + +Each workflow gets a unique webhook URL. Sending an HTTP POST to this URL with a JSON body starts the workflow with that data as input. You can add a webhook secret to verify that incoming requests are genuine. + +#### Workflow Configuration + +Navigate to a workflow's Configuration tab to adjust: + +- Active toggle: enable or disable the workflow. Draft workflows cannot be activated until they are published first. +- Timeout: maximum time in milliseconds a workflow is allowed to run before being stopped. Default is 300,000 ms (5 minutes). +- Max retries: how many times a failed step will be retried before the whole workflow fails. Default is 3. +- Backoff: delay in milliseconds between retry attempts. Default is 1,000 ms. +- Variables: a JSON object of key-value pairs that are available to all steps as shared configuration data. + +#### Testing a Workflow + +Use the Test panel, available from the side panel in the workflow editor, to: + +- Dry Run: shows which steps would run and in what order, without actually executing anything +- Execute: triggers a real run with test input data. Check the Executions tab to see the result + +#### Execution History + +Navigate to the Executions tab of any workflow to see a log of all past runs, including start time, duration, status, and the input and output data at each step. + +### Custom Agents + +Custom Agents are specialized AI assistants you configure for specific tasks. Unlike the default chat agent, which is general-purpose, a custom agent has its own instructions, a defined set of knowledge it can access, a specific AI model, and optional tool restrictions. + +#### Creating a Custom Agent + +1. Navigate to Custom Agents in the sidebar. +2. Click New Agent. +3. Enter a Display Name shown in the agent selector and an Internal Name, a URL-safe slug used in API calls such as `support-agent`. +4. Optionally add a description, then click Create. +5. You will land on the agent configuration page where you can set up its Instructions, Knowledge, Tools, and Webhook. + +#### Instructions Tab + +This is the most important tab. It defines what the agent knows, how it behaves, and what it can do. + +- System Instructions: the prompt sent to the model before every conversation. Use this to define the agent's role, tone, what topics it should and should not cover, and how it should format its answers. +- Model Preset: choose between Fast, Standard, and Advanced. Your admin's model configuration determines which models are in each tier. +- File Preprocessing: when on, attached files are processed with vision AI before being passed to the agent, which helps with image-heavy or scanned documents. +- Structured Responses: when on, the agent formats its answers with consistent structure such as sections and lists instead of free-form text. + +Changes on this tab are saved automatically. A save indicator in the top-right shows the current status. + +#### Knowledge Tab + +Controls which parts of the knowledge base this agent can access. By default, agents can search all organization knowledge. You can restrict it to specific document folders, product categories, or team-scoped data. + +#### Tools Tab + +Controls which platform capabilities the agent can use. Toggle individual tools on or off. For example, a support-only agent might have web browsing turned off but customer lookup turned on. + +#### Webhook Tab + +Each custom agent gets a unique webhook endpoint. You can POST a message and conversation context to this URL to get a response from the agent without using the platform UI. This is useful for integrating the agent into external products or chat widgets. + +#### Versioning + +Custom agents support versioning. When you edit an agent's instructions, a draft version is created. The live version keeps serving requests until you publish the draft. The version history dialog shows all past versions and lets you compare or roll back. + +### Settings + +#### Organization Settings + +Accessible to Admins only. Configure organization-wide settings and manage team members. + +##### Organization Name + +The organization name appears in the sidebar, emails, and notification headers. Edit the name field and click Save Changes to update it. + +##### Member Management + +The member table lists all users in the organization with their email, display name, role, and join date. Admins can: + +- Add members: enter an email, optional password, display name, and role. If the email already exists in Tale, the user is added to the organization without creating a new account. +- Edit members: change a member's display name, role, or set a new password for them. +- Remove members: removes the member from the organization. Their account is not deleted. + +#### Teams + +Teams let you organize members and control knowledge base visibility. A document or conversation tagged to a team only appears when that team is active in the filter. Teams are managed from the Teams settings tab, Admin only. + +#### Integrations + +Configure connections to external services. Tale supports the following integration types: + +##### REST API Integrations + +Connect any HTTP-based API by entering the base URL and credentials. Supported authentication methods: + +- API Key: pass a key in a header or query parameter +- Bearer Token: `Authorization: Bearer ` +- Basic Auth: username and password +- OAuth 2.0: authorization code flow with automatic token refresh + +##### SQL Integrations + +Connect a PostgreSQL or MySQL database. The AI agent and automations can query it using plain language that is translated to SQL. + +##### Email (Conversations) + +Connect an email account to enable the Conversations inbox. Incoming emails become conversation threads. Replies sent from the platform are delivered as normal email responses. + +##### Microsoft OneDrive + +Connect a Microsoft account to enable OneDrive document sync. Users can then import files from OneDrive directly into the Knowledge Base. + +#### API Keys + +Generate API keys for programmatic access to the Tale API. Keys have the same permissions as the user who created them, scoped to their role. Keys can be revoked at any time from this tab. + +#### Branding + +Customize the look of the platform for your organization. Admin only. Available options: + +- Application Name: replaces "Tale" in the browser tab and header +- Logo: upload an image to replace the default logo in the navigation bar +- Favicon: upload a custom favicon for the browser tab, with separate images for light and dark mode +- Brand Color: the primary color used for buttons and active states +- Accent Color: secondary color used for highlights and badges + +#### Account Settings + +Available to all users. Change your password here. If you signed up via SSO, you can also set a regular password from this page to enable direct login. + +#### Audit Logs + +Admin only. A time-ordered record of significant actions taken in the organization, including member changes, integration updates, workflow publications, and settings changes. Useful for compliance and troubleshooting. + +## Roles and Permissions + +Tale uses five roles. Every user belongs to exactly one role within an organization. The same person can have different roles in different organizations. + +### Role Overview + +| Role | Who It's For | +| --- | --- | +| Admin | Full control over the organization. Manages members, settings, integrations, and all content. | +| Developer | For engineers and integration builders. Full data access but cannot manage members or organization settings. | +| Editor | For content and customer service staff. Creates knowledge base content, handles conversations, and approves actions. | +| Member | Read-only access. Can use AI chat to explore data but cannot create or edit content. | + +### Permission Matrix + +#### AI Chat + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| Create and send messages | βœ“ | βœ“ | βœ“ | βœ“ | +| View own chat history | βœ“ | βœ“ | βœ“ | βœ“ | +| Delete and rename conversations | βœ“ | βœ“ | βœ“ | βœ“ | +| Select custom agent | βœ“ | βœ“ | βœ“ | βœ“ | + +#### Knowledge Base + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View all knowledge items | βœ“ | βœ“ | βœ“ | βœ“ | +| Upload / edit / delete documents | β€” | βœ“ | βœ“ | βœ“ | +| Manage products, customers, vendors | β€” | βœ“ | βœ“ | βœ“ | +| Add and configure website crawling | β€” | βœ“ | βœ“ | βœ“ | + +#### Conversations + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View conversations | βœ“ | βœ“ | βœ“ | βœ“ | +| Reply to customers | β€” | βœ“ | βœ“ | βœ“ | +| Close / reopen / archive conversations | β€” | βœ“ | βœ“ | βœ“ | +| Mark as spam | β€” | βœ“ | βœ“ | βœ“ | + +#### Approvals + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View pending approvals | βœ“ | βœ“ | βœ“ | βœ“ | +| Approve or reject actions | β€” | βœ“ | βœ“ | βœ“ | + +#### Automations and Custom Agents + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View automation list | β€” | β€” | βœ“ | βœ“ | +| Create and edit automations | β€” | β€” | βœ“ | βœ“ | +| Publish and activate automations | β€” | β€” | βœ“ | βœ“ | +| View execution logs | β€” | β€” | βœ“ | βœ“ | + +#### Integrations and API + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View integrations | β€” | β€” | βœ“ | βœ“ | +| Configure integrations | β€” | β€” | βœ“ | βœ“ | +| Generate and revoke API keys | β€” | β€” | βœ“ | βœ“ | + +#### Organization Administration + +| Feature | Member | Editor | Developer | Admin | +| --- | --- | --- | --- | --- | +| View organization settings | β€” | β€” | β€” | βœ“ | +| Edit organization name and branding | β€” | β€” | β€” | βœ“ | +| Add and remove members | β€” | β€” | β€” | βœ“ | +| Change member roles | β€” | β€” | β€” | βœ“ | + +### SSO Configuration + +Tale supports Microsoft Entra ID, formerly Azure AD, as a single sign-on provider, so your users can log in with their existing Microsoft accounts. + +#### Enabling SSO + +1. Go to Settings > Integrations. +2. Find the SSO section and enter your Entra ID details: client ID, tenant ID, and client secret. +3. Add the redirect URI in your Azure app registration: `https://yourdomain.com/http_api/api/sso/callback` +4. Save and test the connection. The SSO button will appear on the login page. + +> **Note:** SSO and password login can be used at the same time. Users who joined before SSO was set up can keep using their passwords. Users who sign in via SSO can optionally add a password from Account Settings. + +## API Reference + +Each Tale service has its own REST API. These are used internally between services but are also available for direct integration with external systems. + +### Interactive API Documentation + +All Python-based services have a Swagger UI for exploring and testing the API: + +| Service | Swagger UI URL | OpenAPI JSON | +| --- | --- | --- | +| RAG | http://localhost:8001/docs | http://localhost:8001/openapi.json | +| Crawler | http://localhost:8002/docs | http://localhost:8002/openapi.json | +| Operator | http://localhost:8004/docs | http://localhost:8004/openapi.json | + +### RAG API + +The RAG API handles document indexing and search. It is the engine behind the knowledge base. + +#### Add a Document + +```http +POST /api/v1/documents +``` + +```json +{ + "content": "Your document text here...", + "document_id": "optional-custom-id", + "team_ids": ["team-abc123"], + "metadata": { "source": "manual", "category": "policy" } +} +``` + +Document indexing runs in the background. The response includes a `job_id` you can use to check progress. + +#### Upload a File + +```http +POST /api/v1/documents/upload +Content-Type: multipart/form-data +``` + +```text +file: +team_ids: "team-abc123" +metadata: '{"source": "upload"}' (optional JSON string) +``` + +#### Search the Knowledge Base + +```http +POST /api/v1/search +``` + +```json +{ + "query": "What is our return policy?", + "team_ids": ["team-abc123"], + "top_k": 5, + "similarity_threshold": 0.0 +} +``` + +#### Check Indexing Job Status + +```http +GET /api/v1/jobs/{job_id} +``` + +Job states: `queued`, `running`, `completed`, `failed`. Keep checking this endpoint until the state is `completed` or `failed`. + +### Crawler API + +#### Register a Website for Crawling + +```http +POST /api/v1/websites +``` + +```json +{ + "domain": "https://docs.example.com", + "scan_interval": 21600 +} +``` + +`scan_interval` is in seconds. Minimum value is 60. + +#### Fetch Page Content + +```http +POST /api/v1/urls/fetch +``` + +```json +{ + "urls": ["https://docs.example.com/guide"], + "word_count_threshold": 100 +} +``` + +Returns cached content when available, or fetches it live if not. + +### Platform API + +The Platform service exposes a public API at `/api/v1/*` for programmatic access to your data. Authenticate using a Bearer token from Settings > API Keys. + +Full API documentation: `https://yourdomain.com/api/v1/openapi.json` + +## Operations + +### Monitoring + +All Tale services expose a Prometheus `/metrics` endpoint on the internal Docker network. To enable access from outside, set a bearer token in your `.env` file: + +```dotenv +METRICS_BEARER_TOKEN=your-secret-token-here +``` + +Metrics are then available at these endpoints: + +| Service | Metrics Endpoint | +| --- | --- | +| Crawler | `https://yourdomain.com/metrics/crawler` | +| RAG | `https://yourdomain.com/metrics/rag` | +| Operator | `https://yourdomain.com/metrics/operator` | +| Platform (Express) | `https://yourdomain.com/metrics/platform` | + +> **Note:** The Convex backend exposes over 260 built-in metrics covering query latency, mutation throughput, and scheduler performance. + +### Error Tracking + +Tale supports Sentry and compatible alternatives such as GlitchTip for error tracking. Set your DSN in `.env`: + +```dotenv +SENTRY_DSN=https://your-key@your-sentry-host/project-id +``` + +If `SENTRY_DSN` is not set, error tracking is off and errors only appear in Docker logs. + +### Viewing Logs + +All service logs go to Docker stdout with automatic rotation at 10 MB per file, keeping 3 files per service. + +```bash +# Stream all service logs +docker compose logs -f + +# Stream logs for a specific service +docker compose logs -f rag + +# View recent logs without streaming +docker compose logs --tail=100 platform +``` + +### Database Backups + +To create a database snapshot: + +```bash +docker exec tale-db pg_dump -U tale tale > backup-$(date +%Y%m%d).sql +``` + +To restore from a backup: + +```bash +docker exec -i tale-db psql -U tale tale < backup-20260101.sql +``` + +### Health Checks + +Each service has a health check endpoint: + +| Endpoint | What It Checks | +| --- | --- | +| `GET /health` | Proxy is running and listening | +| `GET /api/health` | Platform is up and Convex backend is reachable | +| `http://localhost:8001/health` | RAG service is running and database pool is connected | +| `http://localhost:8002/health` | Crawler service and browser engine are ready | +| `http://localhost:8004/health` | Operator service is running | + +## Troubleshooting + +### Common Issues + +#### "Docker Engine not found" on Windows + +This means Docker Desktop is not running. Open Docker Desktop from the Start menu or system tray, wait for the engine to show green, then try your command again. + +#### Browser Shows Certificate Warning + +Tale uses a self-signed certificate for local development. You can click through the browser warning or remove it permanently by running: + +```bash +docker exec tale-proxy caddy trust +``` + +Then restart your browser. + +#### Platform Does Not Load After `docker compose up` + +Wait for the platform ready message in the logs. This can take up to two minutes. The `200 OK` health check messages that appear before it do not mean the UI is ready. + +#### AI Responses Are Slow or Failing + +Check your `OPENAI_API_KEY` in `.env`. Common causes: + +- Expired or revoked API key. Regenerate it at openrouter.ai. +- Insufficient credits on your OpenRouter account. +- The model set in `OPENAI_MODEL` is not available on your account tier. +- Network issue between the Tale server and the OpenRouter API. + +#### Documents Are Not Searchable After Upload + +Document indexing runs in the background. After uploading, the RAG service extracts text, splits it into chunks, generates embeddings, and writes to the database. Large files such as multi-hundred-page PDFs can take several minutes. Check the status indicator in Knowledge > Documents to see the current state. + +#### Website Crawling Shows No Pages + +After adding a website, the crawler does an initial pass of the homepage and any links it finds. This takes a few minutes depending on site size. If the page count stays at 0, check `docker compose logs crawler` for errors. Common causes are SSL issues on the target site or `robots.txt` blocks. + +#### Forgot Admin Password + +If you are locked out of your admin account, another admin can reset your password from Settings > Organization > member row > Edit > Set Password. If no admins are available, someone with Docker access can use the Convex Dashboard to update the user record directly. + +### Getting Help + +- Logs: `docker compose logs -f` is always the first place to look +- GitHub Issues: https://github.com/tale-project/tale/issues +- Convex Dashboard: useful for inspecting raw data and function logs when debugging backend problems +- API Docs: `http://localhost:8001/docs`, `/8002/docs`, `/8004/docs` diff --git a/docs/workflow-guide.md b/docs/workflow-guide.md deleted file mode 100644 index cf6ebbfb5..000000000 --- a/docs/workflow-guide.md +++ /dev/null @@ -1,1250 +0,0 @@ -# Workflow System - User Guide - -**Build powerful automation workflows with AI, data processing, and customer engagement.** - ---- - -## Table of Contents - -1. [What You Can Build](#what-you-can-build) -2. [Quick Start](#quick-start) -3. [Building Blocks](#building-blocks) -4. [Working with Data](#working-with-data) -5. [Common Patterns](#common-patterns) -6. [Scheduling & Automation](#scheduling--automation) -7. [Best Practices](#best-practices) - ---- - -## How to Use Workflows - -You don't need to write code! There are two main ways to create and manage workflows: - -### 1. πŸ€– Ask the AI Assistant - -Simply describe what you want in natural language: - -```text -"Create a workflow that sends a welcome email to new customers" - -"Build a workflow that finds inactive customers and sends them a re-engagement offer" - -"I need a workflow that uses AI to recommend products based on purchase history" -``` - -The AI will create the workflow structure for you. - -### 2. πŸ–±οΈ Use the Visual Workflow Editor - -The platform includes a visual workflow editor where you can: - -- **View workflows** - See your workflow as a visual flowchart -- **Add steps** - Click to add new steps to your workflow -- **Connect steps** - Draw connections between steps -- **Configure steps** - Edit step settings through a side panel -- **Test workflows** - Run test executions and see results -- **Chat with AI** - Get help building your workflow from the AI assistant - -The visual editor shows: -- Step types with color-coded icons (Trigger, Action, LLM, Condition, Loop) -- Connections between steps -- Step names and descriptions -- Minimap for navigation -- Real-time validation - -**Note:** The examples in this guide show the JSON structure that represents workflows. You don't need to write JSON manually - the visual editor and AI assistant handle this for you. - ---- - -## What You Can Build - -Workflows let you automate complex business processes by connecting simple building blocks. Here are real examples: - -### πŸ€– AI-Powered Customer Support - -Auto-reply to customer conversations: - -- Detect when a conversation needs a reply -- Generate personalized responses using AI -- Match your brand's tone of voice -- Create approval records for review - -### πŸ“Š Smart Product Recommendations - -Suggest products to customers: - -- Analyze customer purchase history -- Use AI to find relevant products -- Generate personalized recommendations -- Send via email or save to database - -### πŸ” Customer Health Monitoring - -Identify at-risk customers: - -- Assess customer status (active/churned/at-risk) -- Analyze engagement patterns -- Trigger retention campaigns -- Update customer records automatically - -### πŸ“„ Content Processing - -Scan and index website content: - -- Fetch website pages -- Extract metadata and content -- Upload to knowledge base (RAG) -- Keep content synchronized - -### πŸ”„ Data Synchronization - -Keep systems in sync: - -- Import customers from external systems -- Sync products and inventory -- Upload documents to search -- Process data in batches - ---- - -## Understanding Workflow Structure - -A workflow is a series of **steps** connected in sequence. Each step does one thing, then passes control to the next step. - -### Visual Flow - -```text -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Trigger β”‚ ← Workflow starts here -β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ - β”‚ - β–Ό -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Action β”‚ ← Do something (get data, send email, etc.) -β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”˜ - β”‚ - β–Ό -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚Condition β”‚ ← Make a decision -β””β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”˜ - β”‚ β”‚ -true false - β”‚ β”‚ - β–Ό β–Ό -β”Œβ”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β” -β”‚Stepβ”‚ β”‚Stepβ”‚ ← Different paths based on condition -β””β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”˜ -``` - -### Key Concepts - -**Steps** - Individual building blocks -- **Trigger** - How the workflow starts -- **Action** - Do something (get data, send email, update records) -- **LLM** - Use AI to analyze or generate content -- **Condition** - Make decisions and branch -- **Loop** - Process multiple items - -**Connections** - How steps link together -- Each step can go to different next steps based on the outcome -- `success` β†’ next step when things work -- `error` β†’ handle failures -- `true`/`false` β†’ branch based on conditions -- `next`/`done` β†’ loop through items - -**Data Flow** - Information moves through your workflow -- Start with input data -- Each step can access previous step outputs -- Use variables to reference data: `{{customer.email}}`, `{{product.name}}` -- Store secrets securely for API keys and passwords - ---- - -## Quick Start - -Let's build your first workflow: **Send a welcome email to new customers** - -### Tell the AI What You Want - -```text -"Create a workflow that sends a welcome email to new customers. -It should get the customer data and send a personalized email." -``` - -### What the AI Creates - -The AI will build a workflow with 3 steps: - -#### Step 1: Trigger - -- Type: Manual trigger -- Purpose: Start the workflow when you provide a customer ID - -#### Step 2: Get Customer - -- Type: Action (customer.get_by_id) -- Purpose: Fetch customer details -- Input: Customer ID from workflow input -- Output: Customer name, email, and other details - -#### Step 3: Send Email - -- Type: Action (email_send) -- Purpose: Send personalized welcome email -- Input: Customer email and name from Step 2 -- Output: Email sent confirmation - -### How It Works - -```text -You provide: customerId = "cust_456" - ↓ -Step 1: Workflow starts - ↓ -Step 2: Gets customer data - β†’ name: "John Doe" - β†’ email: "john@example.com" - ↓ -Step 3: Sends email - β†’ To: john@example.com - β†’ Subject: "Welcome to our platform!" - β†’ Body: "Hi John Doe, welcome aboard!" - ↓ -Done! βœ“ -``` - -### View and Test - -After the AI creates it: -1. **View** the workflow in the visual editor (see the flowchart) -2. **Test** it with a real customer ID -3. **Activate** it when ready -4. **Run** it manually or on a schedule - -That's it! You've created your first workflow without writing any code. - ---- - -## Building Blocks - -Workflows are built from 5 types of steps. Each step does one thing well. - -### 1. 🎯 Trigger - Start Your Workflow - -**What it does:** Defines how the workflow starts - -**When to use:** Every workflow needs exactly one trigger as the first step - -**Example:** -```json -{ - "stepType": "trigger", - "config": { - "type": "manual" - } -} -``` - -**Trigger types:** -- `manual` - Start manually from UI or API -- `schedule` - Run on a schedule (cron) -- `webhook` - Trigger from external system -- `event` - React to system events - -**Output ports:** `success` - ---- - -### 2. πŸ€– LLM - AI Processing - -**What it does:** Uses AI to analyze data, generate content, or make decisions - -**When to use:** - -- Generate personalized content -- Analyze customer data -- Make intelligent decisions -- Extract information from text - -#### Example: Generate product recommendation - -```json -{ - "stepType": "llm", - "config": { - "name": "Product Recommender", - "systemPrompt": "You are a product recommendation expert.", - "userPrompt": "Recommend products for customer: {{steps.get_customer.output.name}}", - "temperature": 0.7, - "maxTokens": 500, - "outputFormat": "json", - "tools": ["product_search", "customer_get"] - } -} -``` - -**Configuration options:** -- `systemPrompt` - Define the AI's role and behavior (required) -- `userPrompt` - The specific task or question (optional) -- `temperature` - Creativity level: 0 (focused) to 1 (creative) -- `maxTokens` - Maximum response length -- `outputFormat` - `"text"` for natural language, `"json"` for structured data -- `tools` - Array of tools the AI can use (e.g., `["customer_get", "product_search"]`) - -**Available tools:** -- Customer: `customer_get`, `customer_search`, `list_customers` -- Product: `product_get`, `product_search`, `list_products` -- Conversation: `conversation_get`, `conversation_search` -- And more... - -**Output ports:** `success`, `error` - ---- - -### 3. ⚑ Condition - Make Decisions - -**What it does:** Branch your workflow based on data - -**When to use:** - -- Check if a value meets criteria -- Route based on customer status -- Handle different scenarios - -#### Example: Check if customer is VIP - -```json -{ - "stepType": "condition", - "config": { - "expression": "{{steps.get_customer.output.totalSpent}} > 1000", - "description": "Is customer VIP?" - }, - "nextSteps": { - "true": "send_vip_offer", - "false": "send_standard_offer" - } -} -``` - -**Expression examples:** -```javascript -// Numbers -"{{customer.age}} >= 18" -"{{order.total}} > 100 && {{order.total}} < 500" - -// Strings -"{{customer.status}} == 'active'" -"{{customer.email}} contains '@gmail.com'" - -// Arrays -"{{customer.tags}} includes 'vip'" -"{{order.items.length}} > 5" - -// Combining conditions -"{{customer.status}} == 'active' && {{customer.totalSpent}} > 1000" -``` - -**Output ports:** `true`, `false` - ---- - -### 4. πŸ”§ Action - Do Something - -**What it does:** Performs operations like sending emails, updating records, calling APIs, or managing data - -**When to use:** Whenever you need to interact with data or external systems - -Actions are the workhorses of your workflow. Each action type provides specific operations for different parts of your system. - -#### Customer Actions - -Manage customer data and relationships: - -- **create** - Create a new customer -- **get_by_id** - Get customer details by ID -- **query** - Search customers with filters (status, source, etc.) -- **update** - Update customer information -- **filter** - Advanced filtering with expressions (e.g., "totalSpent > 1000") - -**Example use cases:** -- "Get customer by ID and check their status" -- "Find all active customers who haven't purchased in 30 days" -- "Update customer status to VIP" - -#### Product Actions - -Manage product catalog: - -- **create** - Create a new product -- **get_by_id** - Get product details by ID -- **query** - Search products with filters (category, status, etc.) -- **update** - Update product information -- **filter** - Advanced filtering with expressions -- **hydrate_fields** - Enrich data with product details - -**Example use cases:** -- "Get product details for recommendations" -- "Find all active products in a category" -- "Update product stock levels" - -#### Conversation Actions - -Manage customer conversations and messages: - -- **create** - Create a new conversation -- **get_by_id** - Get conversation details -- **query** - Search conversations (by status, customer, channel) -- **query_messages** - Get messages in a conversation -- **update** - Update conversation status or metadata -- **create_from_email** - Create conversation from email thread -- **create_from_sent_email** - Create conversation from sent email - -**Example use cases:** -- "Find all open conversations" -- "Get conversation history for a customer" -- "Update conversation status to resolved" - -#### Document Actions - -Manage documents and files: - -- **create** - Create a new document -- **get_by_id** - Get document details -- **query** - Search documents -- **update** - Update document metadata -- **generate_signed_url** - Get temporary download URL - -**Example use cases:** -- "Upload a document for processing" -- "Get signed URL for document download" -- "Query documents by source provider" - -#### Email Provider Actions - -Get email configuration: - -- **get_default** - Get default email provider settings -- **get_imap_credentials** - Get IMAP credentials for email sync - -**Example use cases:** -- "Get email provider for sending messages" -- "Get IMAP credentials for syncing inbox" - -#### RAG (Knowledge Base) Actions - -Upload content to semantic search: - -- **upload_document** - Upload a document to knowledge base -- **upload_text** - Upload text content directly - -**Example use cases:** -- "Index customer documents for AI search" -- "Upload product descriptions to knowledge base" - -#### Approval Actions - -Manage approval workflows: - -- **create_approval** - Create an approval request -- **update_approval_status** - Approve or reject -- **get_approval** - Get approval details -- **list_pending_approvals** - List all pending approvals -- **get_approval_history** - Get approval history - -**Example use cases:** -- "Create approval for AI-generated email" -- "List pending approvals for review" -- "Approve or reject a request" - -#### Integration Actions - -Connect to external systems: - -- **Execute any operation** from configured integrations (Shopify, Circuly, custom APIs, etc.) - -**Example use cases:** -- "Sync orders from Shopify" -- "Get subscription data from Circuly" -- "Call custom ERP API" - -#### Utility Actions - -General-purpose operations: - -- **set_variables** - Store data for later use in the workflow -- **workflow.upload_all_workflows** - Upload workflows to knowledge base - -**Example use cases:** -- "Store calculated values" -- "Save intermediate results" - -**Output ports:** All actions have `success` and `error` ports - ---- - -### 5. πŸ”„ Loop - Process Collections - -**What it does:** Iterate over arrays and process each item - -**When to use:** -- Send emails to multiple customers -- Process a list of products -- Handle batch operations - -**Example: Send email to each customer** -```json -{ - "stepSlug": "loop_customers", - "stepType": "loop", - "config": { - "items": "{{steps.get_customers.output.customers}}", - "itemVariable": "customer" - }, - "nextSteps": { - "next": "send_email", - "done": "finish" - } -} -``` - -**Inside the loop:** -```json -{ - "stepSlug": "send_email", - "stepType": "action", - "config": { - "type": "email_send", - "parameters": { - "to": "{{loop.item.email}}", - "subject": "Hello {{loop.item.name}}!", - "body": "You are customer #{{loop.index}}" - } - }, - "nextSteps": { - "success": "loop_customers" - } -} -``` - -**How it works:** -1. Loop starts with first item -2. Executes the `next` step for that item -3. That step returns to the loop -4. Loop moves to next item -5. Repeats until all items processed -6. Then goes to `done` step - -**Loop variables:** -- `{{loop.item}}` - Current item being processed -- `{{loop.item.propertyName}}` - Access properties of current item -- `{{loop.index}}` - Current position (starts at 0) -- `{{loop.parent.item}}` - For nested loops (access outer loop's item) - -**Output ports:** `next`, `done`, `error` - ---- - -## Working with Data - -### Variables and Templating - -Use `{{variable}}` syntax to access data anywhere in your workflow: - -**Input data:** -```typescript -{{input.customerId}} -{{input.productName}} -``` - -**Step outputs:** -```typescript -{{steps.get_customer.output.email}} -{{steps.analyze.output.score}} -``` - -**Loop context:** -```typescript -{{loop.item.name}} -{{loop.index}} -``` - -**Built-in variables:** -```typescript -{{organizationId}} // Current organization -{{wfDefinitionId}} // Workflow ID -{{$now}} // Current timestamp -``` - -### Using Secrets - -Store sensitive data (API keys, passwords) securely: - -```typescript -// In workflow config -config: { - secrets: { - apiKey: { - kind: 'inlineEncrypted', - cipherText: '...' // Encrypted value - } - } -} - -// Use in steps -parameters: { - authorization: '{{secrets.apiKey}}' -} -``` - -### Accessing Nested Data - -```typescript -// Object properties -{{customer.profile.address.city}} - -// Array items -{{products[0].name}} - -// With filters (Mustache) -{{#customers}} - {{name}}: {{email}} -{{/customers}} -``` - ---- - -## Common Patterns - -### Pattern 1: AI-Powered Customer Engagement - -**Use case:** Automatically reply to customer messages with AI - -**What you tell the AI:** -> "Create a workflow that finds open conversations, uses AI to decide if they need a reply, generates a response, and saves it for approval." - -**Workflow structure:** -```json -{ - "name": "Auto-Reply to Conversations", - "steps": [ - { - "stepSlug": "find_conversations", - "stepType": "action", - "config": { - "type": "conversation_search", - "parameters": { - "status": "open", - "limit": 10 - } - }, - "nextSteps": { "success": "loop_conversations" } - }, - { - "stepSlug": "loop_conversations", - "stepType": "loop", - "config": { - "items": "{{steps.find_conversations.output.conversations}}", - "itemVariable": "conversation" - }, - "nextSteps": { - "next": "check_needs_reply", - "done": "finish" - } - }, - { - "stepSlug": "check_needs_reply", - "stepType": "llm", - "config": { - "systemPrompt": "Analyze if this conversation needs a reply. Return JSON with 'needsReply' boolean.", - "userPrompt": "Conversation: {{loop.item.messages}}", - "outputFormat": "json" - }, - "nextSteps": { "success": "decide_reply" } - }, - { - "stepSlug": "decide_reply", - "stepType": "condition", - "config": { - "expression": "{{steps.check_needs_reply.output.needsReply}} == true" - }, - "nextSteps": { - "true": "generate_reply", - "false": "loop_conversations" - } - }, - { - "stepSlug": "generate_reply", - "stepType": "llm", - "config": { - "systemPrompt": "You are a helpful customer service agent. Generate a professional reply.", - "userPrompt": "Reply to: {{loop.item.lastMessage}}" - }, - "nextSteps": { "success": "save_reply" } - }, - { - "stepSlug": "save_reply", - "stepType": "action", - "config": { - "type": "conversation_create_draft", - "parameters": { - "conversationId": "{{loop.item._id}}", - "message": "{{steps.generate_reply.output}}" - } - }, - "nextSteps": { "success": "loop_conversations" } - } - ] -} -``` - -**What happens:** -1. Finds up to 10 open conversations -2. Loops through each conversation -3. AI analyzes if it needs a reply -4. If yes, AI generates a professional response -5. Saves the response as a draft for human approval -6. Continues to next conversation - ---- - -### Pattern 2: Smart Product Recommendations - -**Use case:** Recommend products based on customer behavior - -**What you tell the AI:** -> "Create a workflow that gets a customer's purchase history, uses AI to recommend 3 relevant products, saves the recommendations, and sends them via email." - -**Workflow structure:** -```json -{ - "name": "Product Recommendations", - "steps": [ - { - "stepSlug": "get_customer", - "stepType": "action", - "config": { - "type": "customer_get", - "parameters": { - "customerId": "{{input.customerId}}" - } - }, - "nextSteps": { "success": "get_orders" } - }, - { - "stepSlug": "get_orders", - "stepType": "action", - "config": { - "type": "customer_orders", - "parameters": { - "customerId": "{{input.customerId}}", - "limit": 20 - } - }, - "nextSteps": { "success": "recommend_products" } - }, - { - "stepSlug": "recommend_products", - "stepType": "llm", - "config": { - "systemPrompt": "You are a product recommendation expert. Analyze purchase history and recommend 3 relevant products.", - "userPrompt": "Customer: {{steps.get_customer.output.name}}\nPurchase history: {{steps.get_orders.output}}", - "outputFormat": "json", - "tools": ["product_search", "product_get"] - }, - "nextSteps": { "success": "save_recommendations" } - }, - { - "stepSlug": "save_recommendations", - "stepType": "action", - "config": { - "type": "recommendation_create", - "parameters": { - "customerId": "{{input.customerId}}", - "products": "{{steps.recommend_products.output.products}}" - } - }, - "nextSteps": { "success": "send_email" } - }, - { - "stepSlug": "send_email", - "stepType": "action", - "config": { - "type": "email_send", - "parameters": { - "to": "{{steps.get_customer.output.email}}", - "subject": "Products you might love", - "body": "Hi {{steps.get_customer.output.name}}, based on your recent purchases, we think you'll love these: {{steps.recommend_products.output.products}}" - } - }, - "nextSteps": { "success": "end" } - } - ] -} -``` - -**What happens:** -1. Gets customer data -2. Fetches their last 20 orders -3. AI analyzes purchase patterns and recommends 3 products (can search product catalog) -4. Saves recommendations to database -5. Sends personalized email with recommendations - ---- - -### Pattern 3: Batch Processing with Loops - -**Use case:** Process multiple items efficiently - -**What you tell the AI:** -> "Create a workflow that finds inactive customers, uses AI to assess their status, and updates their records accordingly." - -**Workflow structure:** -```json -{ - "name": "Assess Customer Health", - "steps": [ - { - "stepSlug": "find_inactive", - "stepType": "action", - "config": { - "type": "customer_search", - "parameters": { - "status": "inactive", - "limit": 100 - } - }, - "nextSteps": { "success": "loop_customers" } - }, - { - "stepSlug": "loop_customers", - "stepType": "loop", - "config": { - "items": "{{steps.find_inactive.output.customers}}", - "itemVariable": "customer" - }, - "nextSteps": { - "next": "assess_customer", - "done": "finish" - } - }, - { - "stepSlug": "assess_customer", - "stepType": "llm", - "config": { - "systemPrompt": "Assess customer health. Return JSON with 'status' field: 'active', 'at_risk', or 'churned'.", - "userPrompt": "Customer data: {{loop.item}}", - "outputFormat": "json" - }, - "nextSteps": { "success": "update_customer" } - }, - { - "stepSlug": "update_customer", - "stepType": "action", - "config": { - "type": "customer_update", - "parameters": { - "customerId": "{{loop.item._id}}", - "status": "{{steps.assess_customer.output.status}}" - } - }, - "nextSteps": { "success": "loop_customers" } - } - ] -} -``` - -**What happens:** -1. Finds up to 100 inactive customers -2. Loops through each customer -3. AI analyzes their data and determines status -4. Updates customer record with new status -5. Continues to next customer - ---- - -### Pattern 4: Multi-Step AI Conversation - -**Use case:** AI steps that build on each other's context - -**What you tell the AI:** -> "Create a workflow where AI first analyzes customer data, then based on that analysis, recommends specific actions. The AI should remember the first analysis when making recommendations." - -**Important:** To share context between AI steps, you need to provide a `threadId` when starting the workflow. - -**Workflow structure:** -```json -{ - "name": "Customer Analysis with Recommendations", - "steps": [ - { - "stepSlug": "get_customer", - "stepType": "action", - "config": { - "type": "customer_get", - "parameters": { - "customerId": "{{input.customerId}}" - } - }, - "nextSteps": { "success": "analyze_customer" } - }, - { - "stepSlug": "analyze_customer", - "stepType": "llm", - "config": { - "systemPrompt": "You are a customer success analyst. Analyze the customer's data and identify patterns.", - "userPrompt": "Analyze this customer: {{steps.get_customer.output}}" - }, - "nextSteps": { "success": "recommend_actions" } - }, - { - "stepSlug": "recommend_actions", - "stepType": "llm", - "config": { - "systemPrompt": "Based on your previous analysis, recommend 3 specific actions we should take.", - "userPrompt": "What should we do?" - }, - "nextSteps": { "success": "save_recommendations" } - }, - { - "stepSlug": "save_recommendations", - "stepType": "action", - "config": { - "type": "set_variables", - "parameters": { - "analysis": "{{steps.analyze_customer.output}}", - "recommendations": "{{steps.recommend_actions.output}}" - } - }, - "nextSteps": { "success": "end" } - } - ] -} -``` - -**How to run with shared context:** -When you start this workflow, provide a `threadId`: -```json -{ - "input": { - "customerId": "cust_456" - }, - "threadId": "thread_abc123" -} -``` - -**What happens:** -1. Gets customer data -2. AI analyzes the customer (creates analysis in thread) -3. AI recommends actions (remembers the analysis from step 2) -4. Saves both analysis and recommendations - -**Without threadId:** Each AI step would start fresh with no memory of previous steps. - ---- - -### Pattern 5: Error Handling - -**Use case:** Handle failures gracefully - -**What you tell the AI:** -> "Create a workflow that sends an email to a customer. If it succeeds, mark them as contacted. If it fails, log the error for review." - -**Workflow structure:** -```json -{ - "name": "Send Email with Error Handling", - "steps": [ - { - "stepSlug": "get_customer", - "stepType": "action", - "config": { - "type": "customer_get", - "parameters": { - "customerId": "{{input.customerId}}" - } - }, - "nextSteps": { "success": "send_email" } - }, - { - "stepSlug": "send_email", - "stepType": "action", - "config": { - "type": "email_send", - "parameters": { - "to": "{{steps.get_customer.output.email}}", - "subject": "Important update", - "body": "Hi {{steps.get_customer.output.name}}, ..." - } - }, - "nextSteps": { - "success": "mark_sent", - "error": "log_failure" - } - }, - { - "stepSlug": "mark_sent", - "stepType": "action", - "config": { - "type": "customer_update", - "parameters": { - "customerId": "{{input.customerId}}", - "emailSent": true, - "lastContactedAt": "{{$now}}" - } - }, - "nextSteps": { "success": "end" } - }, - { - "stepSlug": "log_failure", - "stepType": "action", - "config": { - "type": "set_variables", - "parameters": { - "errorMessage": "{{steps.send_email.error}}", - "failedAt": "{{$now}}", - "customerId": "{{input.customerId}}" - } - }, - "nextSteps": { "success": "end" } - } - ] -} -``` - -**What happens:** - -**Success path:** -1. Gets customer data -2. Sends email successfully -3. Updates customer record with `emailSent: true` - -**Error path:** -1. Gets customer data -2. Email fails (invalid address, service down, etc.) -3. Logs error details for review - ---- - -## Scheduling & Automation - -### Run on a Schedule - -Use cron expressions to run workflows automatically: - -```typescript -// In workflow config -config: { - schedule: { - cron: '0 9 * * *', // Every day at 9 AM - timezone: 'America/New_York' - } -} -``` - -**Common schedules:** -- `0 * * * *` - Every hour -- `0 9 * * *` - Daily at 9 AM -- `0 9 * * 1` - Every Monday at 9 AM -- `*/15 * * * *` - Every 15 minutes -- `0 0 1 * *` - First day of each month - -### Trigger from Events - -```typescript -{ - stepType: 'trigger', - config: { - type: 'event', - eventType: 'customer.created' - } -} -``` - -### Webhook Triggers - -```typescript -{ - stepType: 'trigger', - config: { - type: 'webhook', - path: '/webhooks/new-order' - } -} -``` - ---- - -## Best Practices - -### βœ… Do's - -#### 1. Keep steps focused - -- Each step should do one thing well -- Break complex logic into multiple steps - -#### 2. Use descriptive names - -```typescript -// Good -stepSlug: 'check_customer_vip_status' - -// Bad -stepSlug: 'step3' -``` - -#### 3. Handle errors - -- Always define `error` ports for actions and LLM steps -- Log failures for debugging - -#### 4. Test with small data first - -- Start with 1-5 items in loops -- Scale up after testing - -#### 5. Use variables wisely - -- Store reusable values in variables -- Keep secrets encrypted - -#### 6. Add descriptions - -```typescript -{ - name: 'Send Welcome Email', - description: 'Sends personalized welcome email to new customers within 24 hours' -} -``` - -### ❌ Don'ts - -#### 1. Don't create circular flows - -```typescript -// Bad: Infinite loop -step1 β†’ step2 β†’ step1 -``` - -#### 2. Don't hardcode sensitive data - -```typescript -// Bad -apiKey: 'your-api-key' - -// Good -apiKey: '{{secrets.apiKey}}' -``` - -#### 3. Don't process huge batches in one workflow - -```typescript -// Bad: 10,000 items in one loop -items: allCustomers // 10,000 customers - -// Good: Process in chunks -items: '{{steps.get_batch.output.customers}}' // 100 at a time -``` - -#### 4. Don't skip error handling - -```typescript -// Bad: No error port -nextSteps: { success: 'next_step' } - -// Good: Handle both cases -nextSteps: { - success: 'next_step', - error: 'log_error' -} -``` - -#### 5. Don't nest loops too deeply - -```typescript -// Bad: 3+ levels of nesting -loop β†’ loop β†’ loop β†’ loop - -// Good: Flatten or use separate workflows -``` - ---- - -## Debugging Tips - -### View Execution Logs - -```typescript -// Get execution details -const execution = await ctx.runQuery(api.wf_executions.getExecution, { - executionId: 'exec_123' -}); - -console.log(execution.status); // 'running', 'completed', 'failed' -console.log(execution.currentStepSlug); // Which step is running -console.log(execution.variables); // Current variable state -``` - -### Common Issues - -#### Issue: "Variable not found" - -```typescript -// Check variable path -{{steps.get_customer.output.email}} // βœ… Correct -{{customer.email}} // ❌ Wrong - missing 'steps.' -``` - -#### Issue: "Loop not iterating" - -```typescript -// Make sure to return to loop -nextSteps: { - success: 'loop_continue' // Must go back to loop step -} -``` - -#### Issue: "AI not using tools" - -```typescript -// Make sure tools are specified -config: { - tools: ['customer_get', 'product_search'], // βœ… Specify tools - systemPrompt: 'Use tools to find customer data' -} -``` - -#### Issue: "Workflow stuck" - -- Check for missing `nextSteps` definitions -- Verify all ports are connected -- Look for condition expressions that never evaluate to true - ---- - -## Next Steps - -Ready to build your first workflow? Here's what to do: - -1. **Start Simple** - Begin with a basic workflow like "send welcome email" -2. **Ask the AI** - Describe what you want in natural language -3. **Test It** - Run your workflow with test data -4. **Iterate** - Add more steps and complexity as needed -5. **Explore Examples** - Check `examples/workflows/` for complete working examples - ---- - -## Need Help? - -- **Ask the AI Assistant** - It can help you build and debug workflows -- **Check Execution Logs** - See what happened at each step -- **Test with Small Data** - Start with 1-5 items before scaling up -- **Start Simple** - Build basic workflows first, then add complexity - -Happy automating! πŸš€ - - - - - diff --git a/docs/zero-downtime-deployment.md b/docs/zero-downtime-deployment.md deleted file mode 100644 index d19ecacb4..000000000 --- a/docs/zero-downtime-deployment.md +++ /dev/null @@ -1,231 +0,0 @@ -# Zero-Downtime Deployment - -This guide covers blue-green deployment for production environments where downtime is not acceptable. - -> **Looking for simple deployment?** See the main [README](../README.md) for standard Docker Compose deployment. - -## Overview - -Blue-green deployment runs two versions of stateless services simultaneously: - -1. **Blue** and **Green** environments alternate as active/standby -2. New version starts alongside the current one -3. Traffic switches only after health checks pass -4. Old version is drained and removed - -**Stateful services** (database, proxy) are shared between environments and are not affected by deployments. - -## Requirements - -- Docker and Docker Compose -- At least **12-16 GB RAM** (runs 2x services during deployment) -- Services must respond to health checks within 5 minutes - -## Quick Start - -```bash -# Deploy specific version (zero-downtime) -./scripts/deploy.sh deploy v1.0.0 - -# Deploy latest version -./scripts/deploy.sh deploy latest - -# Check current status -./scripts/deploy.sh status - -# Rollback to previous version -./scripts/deploy.sh rollback - -# Clean up inactive containers -./scripts/deploy.sh cleanup -``` - -## Commands - -### deploy.sh - -| Command | Description | -|---------|-------------| -| `deploy ` | Pull and deploy specified version with zero downtime | -| `status` | Show current deployment status and health | -| `rollback` | Revert to the previous version | -| `cleanup` | Remove inactive containers | -| `reset` | Remove ALL blue-green containers and return to normal mode | -| `help` | Show usage information | - - -## Configuration - -Environment variables to customize deployment: - -| Variable | Default | Description | -|----------|---------|-------------| -| `HEALTH_CHECK_TIMEOUT` | 300s | Max time to wait for health checks | -| `HEALTH_CHECK_INTERVAL` | 3s | Interval between health checks | -| `DRAIN_TIMEOUT` | 30s | Time to drain old containers before removal | - -Example with custom timeout: - -```bash -HEALTH_CHECK_TIMEOUT=300 ./scripts/deploy.sh deploy v1.0.0 -``` - -## How It Works - -### Architecture - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Caddy Proxy β”‚ -β”‚ (Routes traffic to healthy backend) β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β–Ό β–Ό -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Blue Services β”‚ β”‚ Green Services β”‚ -β”‚ (Active) β”‚ β”‚ (Standby) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ platform-blue β”‚ β”‚ platform-green β”‚ -β”‚ rag-blue β”‚ β”‚ rag-green β”‚ -β”‚ crawler-blue β”‚ β”‚ crawler-green β”‚ -β”‚ operator-blue β”‚ β”‚ operator-green β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β–Ό - β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” - β”‚ Shared Services β”‚ - β”‚ (Stateful) β”‚ - β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ - β”‚ db (TimescaleDB) β”‚ - β”‚ proxy (Caddy) β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -### Deployment Flow - -1. **Pull images** - Download specified version from GHCR -2. **Detect current state** - Determine which color (blue/green) is currently active -3. **Start new version** - Run the new containers alongside existing ones -4. **Health checks** - Wait for all new services to report healthy -5. **Traffic switch** - Caddy automatically routes to healthy backends -6. **Drain old version** - Wait for in-flight requests to complete -7. **Cleanup** - Remove old containers - -### Files - -| File | Purpose | -|------|---------| -| `compose.yml` | Base configuration for all services | -| `compose.blue.yml` | Blue environment overlay (container names, network aliases) | -| `compose.green.yml` | Green environment overlay | -| `scripts/deploy.sh` | Deployment automation script | -| `.deployment-color` | Tracks current active deployment (auto-generated) | - -## Database Migrations - -Database changes require special handling since the database is shared between blue and green environments. Both old and new versions of the application must work with the database during the transition period. - -### Guidelines - -1. **Backward-compatible migrations only** - New code must work with old schema, old code must work with new schema -2. **Expand-contract pattern** - Split breaking changes into multiple deployments: - - **Deploy 1 (Expand)**: Add new columns/tables, keep old ones - - **Deploy 2 (Migrate)**: Application uses new structure - - **Deploy 3 (Contract)**: Remove old columns/tables after confirming success - -### Examples - -| Change | Safe Approach | -|--------|---------------| -| Add column | Add as nullable or with default value | -| Remove column | First deploy code that stops using it, then remove | -| Rename column | Add new column β†’ migrate data β†’ update code β†’ remove old | -| Change type | Add new column with new type β†’ migrate β†’ update code β†’ remove old | - -### Migration Workflow - -```bash -# 1. Run migrations BEFORE deploying new code -bun run db:migrate # or your migration command - -# 2. Deploy new version -./scripts/deploy.sh deploy v1.0.0 - -# 3. Verify everything works - -# 4. (Later) Run cleanup migrations if needed -bun run db:migrate:cleanup -``` - -### Non-Backward-Compatible Changes - -If a migration cannot be made backward-compatible: - -1. Schedule a maintenance window -2. Use standard deployment with downtime -3. Or coordinate a multi-phase rollout over several deployments - -## Rollback - -If a deployment fails or you need to revert: - -```bash -./scripts/deploy.sh rollback -``` - -Rollback restarts containers from the previous deployment. It works best immediately after a failed deployment when the previous containers still exist. - -For a fresh deployment after cleanup: - -```bash -./scripts/deploy.sh deploy v1.0.0 -``` - -## Troubleshooting - -### Health checks timing out - -```bash -# Increase timeout -HEALTH_CHECK_TIMEOUT=300 ./scripts/deploy.sh deploy v1.0.0 - -``` - -### Deployment failed - -```bash -# Clean up failed deployment -./scripts/deploy.sh cleanup - -# Try again -./scripts/deploy.sh deploy v1.0.0 -``` - -### Check service health - -```bash -# View all container status -docker ps --format "table {{.Names}}\t{{.Status}}" - -# Check specific service -docker inspect --format='{{.State.Health.Status}}' tale-platform-blue -``` - -### Memory issues during deployment - -Blue-green deployment temporarily runs 2x services. If you run out of memory: - -1. Increase available RAM to at least 12-16 GB -2. Or use standard deployment instead (with brief downtime) - -## When to Use - -| Scenario | Recommendation | -|----------|----------------| -| Development / Testing | Standard deployment | -| Low traffic, scheduled maintenance OK | Standard deployment | -| Production, zero-downtime required | Blue-green deployment | -| High availability requirements | Blue-green deployment | -| Limited server resources (`<12 GB RAM`) | Standard deployment |