Feature: Self-Contained Agents (Custom GPTs-like)

[larryro]

Summary

Implement user-configurable "self-contained agents" - bundled configurations of model + knowledge + system prompt + tools that can be created, shared, and used by teams within an organization.

Similar to: ChatGPT Custom GPTs, OpenWebUI Workspaces, GitHub Copilot Custom Agents

Design Decisions

Decision	Choice
Integration	Users can select any agent directly in chat UI (routing agent OR specific agents)
Sharing	Organization-internal only (no marketplace)
Versioning	Required for MVP - full version history with rollback
Model Selection	Predefined functional presets (e.g., "Fast", "Standard", "Advanced")

Key Deliverables

1. Data Model: customAgents and customAgentVersions tables
2. Backend: Custom agent factory, CRUD operations, execution integration
3. Frontend: Agent management UI, chat agent selector, tool picker
4. Versioning: Full version history with rollback capability

See detailed implementation plan in the comments below.

[larryro]

Self-Contained Agents Feature Plan

Overview

Implement user-configurable "self-contained agents" — bundled configurations of model + knowledge + system prompt + tools that can be created, shared, and used by teams within an organization.

Similar to: ChatGPT Custom GPTs, OpenWebUI Workspaces, GitHub Copilot Custom Agents

---

Design Decisions

Decision	Choice
Integration	Users can select any agent directly in chat UI (routing agent OR specific agents)
Sharing	Organization-internal only (no marketplace)
Versioning	Required for MVP — full version history with rollback
Model Selection	Predefined functional presets (e.g., "Fast", "Standard", "Advanced")
Execution	Reuse existing chat agent pipeline (startAgentChat → generateAgentResponse)
Team Architecture	Plan B: Team as Context — single teamId ownership, global team filter in UI, knowledge auto-scoped to team

---

Team Architecture

Decision: Team as Context (Plan B)

New modules (custom agents, and future team-scoped resources) use a single teamId field for team ownership. A global team filter in the UI controls which resources the user sees.

teamId: null           → org-wide (visible to all members)
teamId: 'team-sales'   → belongs to Sales team (visible when user is in Sales or "All Teams")

How the global team filter works:

• User selects "All Teams" → sees org-wide resources + resources from all their teams
• User selects "Sales" → sees org-wide resources + Sales resources only
• Visibility is automatic — no per-resource visibility field needed

Knowledge scope is automatic: When an agent belongs to a team, its RAG search is automatically narrowed to that team's knowledge. No separate knowledgeScopeMode field needed.

Unified Team Fields

All team-scoped tables (including documents after migration) use the same two fields:

// Standard team fields for all team-scoped tables
teamId: v.optional(v.string()),                    // Primary team (null = org-wide)
sharedWithTeamIds: v.optional(v.array(v.string())) // Additional teams with access

Shared access helper used by all modules:

// convex/lib/team_access.ts

function hasTeamAccess(
  resource: { teamId?: string | null; sharedWithTeamIds?: string[] },
  userTeamIds: string[],
): boolean {
  if (!resource.teamId) return true; // org-wide
  if (userTeamIds.includes(resource.teamId)) return true; // primary team
  return resource.sharedWithTeamIds?.some(id => userTeamIds.includes(id)) ?? false;
}

Documents Migration (teamTags → teamId + sharedWithTeamIds)

Documents currently use teamTags: string[]. We migrate to the unified model:

Current teamTags	→ teamId	→ sharedWithTeamIds
undefined / []	null	undefined
['sales']	'sales'	undefined
['sales', 'marketing']	'sales'	['marketing']

Migration steps (additive, with transition period):

1. Add teamId + sharedWithTeamIds fields to documents schema (alongside existing teamTags)
2. Run migration script to populate new fields from teamTags
3. Update get_documents_cursor.ts and rls_rules.ts to use new fields
4. Verify no regressions
5. Remove teamTags field

// Migration script (batch mutation)
for await (const doc of ctx.db.query('documents')) {
  const tags = doc.teamTags ?? [];
  await ctx.db.patch(doc._id, {
    teamId: tags[0] ?? null,
    sharedWithTeamIds: tags.length > 1 ? tags.slice(1) : undefined,
  });
}

Before running migration: Query data distribution to validate assumptions:

// How many documents have 2+ team tags?
const multi = docs.filter(d => (d.teamTags?.length ?? 0) > 1);

---

Architecture: Reuse Strategy

The existing codebase already provides the full execution pipeline used by all 6 built-in agents. Custom agents plug into it via a thin conversion layer — no new execution engine needed.

┌─────────────────────────────────────┐
│  customAgents table (NEW)           │  ← Store config in DB
│  → SerializableAgentConfig (EXISTS) │  ← Thin conversion layer
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│  chatWithCustomAgent mutation (NEW)  │  ← New entry point
│  → startAgentChat() (EXISTS)        │  ← Reuse unified mutation helper
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│  runAgentGeneration (EXISTS)         │  ← Reuse internal action
│  → createAgentConfig() (EXISTS)     │  ← Reuse agent factory
│  → generateAgentResponse() (EXISTS)  │  ← Reuse execution engine
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│  Agent SDK + Tool Registry (EXISTS)  │  ← Shared foundation
└─────────────────────────────────────┘

Existing components reused directly

Component	File	Role
SerializableAgentConfig	convex/lib/agent_chat/types.ts	Target type for DB → config conversion
createAgentConfig()	convex/lib/create_agent_config.ts	Agent factory (model, tools, instructions)
startAgentChat()	convex/lib/agent_chat/start_agent_chat.ts	Unified mutation helper (message dedup, streaming, scheduling)
generateAgentResponse()	convex/lib/agent_response/generate_response.ts	Execution engine (streaming, hooks, context management)
Tool Registry	convex/agent_tools/tool_registry.ts	27+ tools selectable by name
AGENT_CONTEXT_CONFIGS	convex/lib/context_management/constants.ts	Context budget per agent type
getDefaultAgentRuntimeConfig()	convex/lib/agent_runtime_config.ts	Model/provider resolution
Hook system (AgentHooksConfig)	convex/lib/agent_chat/types.ts	beforeContext, beforeGenerate, afterGenerate
getUserTeamIds()	convex/lib/get_user_teams.ts	Resolve user's team memberships

New code required

Component	Purpose
customAgents + customAgentVersions tables	Persistent agent configurations
toSerializableConfig()	DB record → SerializableAgentConfig conversion (thin mapping)
chatWithCustomAgent mutation	Entry point: load from DB → call startAgentChat()
hasTeamAccess()	Shared team access helper (teamId + sharedWithTeamIds)
Documents teamTags migration	Migrate existing documents to unified team fields
AgentType extension	Add 'custom' to the type union + context config
CRUD mutations/queries	Create, update, delete, duplicate, list, version management
Frontend management UI	Agent editor, tool picker, chat selector, global team filter

---

Current State

Tale already has:

• 6 hardcoded agents (chat/routing, CRM, document, web, workflow, integration), all following the same SerializableAgentConfig → startAgentChat() → generateAgentResponse() pattern
• 27 tools in a centralized registry (convex/agent_tools/tool_registry.ts)
• Hook system for context injection (beforeContext, beforeGenerate, onError)
• Knowledge base with RAG/vector search (documents, websites, products, customers)
• Tone of Voice per organization
• Team-based data isolation for knowledge access (userTeamIds in RAG)
• Environment-based model selection with preset support

What's missing:

• Database-stored agent configurations
• Per-agent model selection (via presets)
• Global team context in UI (teams exist but have no active selection)
• UI for creating/editing agents
• Tool picker interface

---

Data Model

New Table: customAgents

// convex/schema.ts
customAgents: defineTable({
  organizationId: v.id('organizations'),

  // Identity
  name: v.string(),                    // Internal identifier (slug)
  displayName: v.string(),             // Human-readable name
  description: v.optional(v.string()), // What this agent does
  avatarUrl: v.optional(v.string()),   // Agent icon/avatar

  // Configuration (mirrors SerializableAgentConfig fields)
  systemInstructions: v.string(),      // → instructions
  toolNames: v.array(v.string()),      // → convexToolNames

  // Model settings (predefined presets)
  modelPreset: v.union(
    v.literal('fast'),      // → useFastModel: true (OPENAI_FAST_MODEL)
    v.literal('standard'),  // → default OPENAI_MODEL
    v.literal('advanced'),  // → model: OPENAI_CODING_MODEL
    v.literal('vision')     // → model: OPENAI_VISION_MODEL
  ),
  temperature: v.optional(v.number()), // 0.0-1.0
  maxTokens: v.optional(v.number()),
  maxSteps: v.optional(v.number()),    // Tool call loop limit

  // Knowledge binding
  includeKnowledge: v.boolean(),       // → enableVectorSearch
  knowledgeTopK: v.optional(v.number()),
  toneOfVoiceId: v.optional(v.id('toneOfVoice')),

  // Team ownership (Plan B: Team as Context — unified fields)
  teamId: v.optional(v.string()),                    // null = org-wide, string = belongs to team
  sharedWithTeamIds: v.optional(v.array(v.string())), // Additional teams with access
  createdBy: v.id('members'),

  // Metadata
  isActive: v.boolean(),
  currentVersion: v.number(),
  metadata: v.optional(v.any()),
})
  .index('by_organization', ['organizationId'])
  .index('by_organization_active', ['organizationId', 'isActive'])
  .index('by_team', ['teamId'])
  .index('by_creator', ['createdBy'])

New Table: customAgentVersions

customAgentVersions: defineTable({
  customAgentId: v.id('customAgents'),
  version: v.number(),

  // Snapshot of configuration at this version
  systemInstructions: v.string(),
  toolNames: v.array(v.string()),
  modelPreset: v.union(
    v.literal('fast'),
    v.literal('standard'),
    v.literal('advanced'),
    v.literal('vision')
  ),
  temperature: v.optional(v.number()),
  maxTokens: v.optional(v.number()),
  maxSteps: v.optional(v.number()),
  includeKnowledge: v.boolean(),
  knowledgeTopK: v.optional(v.number()),

  // Version metadata
  createdAt: v.number(),
  createdBy: v.id('members'),
  changeDescription: v.optional(v.string()),
})
  .index('by_agent', ['customAgentId'])
  .index('by_agent_version', ['customAgentId', 'version'])

Version behavior:

• Every save creates a new version (auto-increment)
• customAgents.currentVersion points to active version
• Users can view history and rollback to any previous version
• Rollback creates a new version (copy of old config), preserving full history

---

Backend Implementation

1. DB Config → SerializableAgentConfig Conversion

Create convex/custom_agents/config.ts — a thin mapping layer:

import { SerializableAgentConfig } from '../lib/agent_chat/types';

export function toSerializableConfig(
  agent: Doc<'customAgents'>,
): SerializableAgentConfig {
  return {
    name: `custom:${agent.name}`,
    instructions: agent.systemInstructions,
    convexToolNames: agent.toolNames as ToolName[],
    useFastModel: agent.modelPreset === 'fast',
    model: resolveModelPreset(agent.modelPreset),
    temperature: agent.temperature,
    maxTokens: agent.maxTokens,
    maxSteps: agent.maxSteps ?? 15,
    enableVectorSearch: agent.includeKnowledge,
  };
}

function resolveModelPreset(preset: string): string | undefined {
  switch (preset) {
    case 'fast': return undefined;    // handled by useFastModel flag
    case 'advanced': return process.env.OPENAI_CODING_MODEL;
    case 'vision': return process.env.OPENAI_VISION_MODEL;
    default: return undefined;        // 'standard' uses default OPENAI_MODEL
  }
}

2. New Mutation: chatWithCustomAgent

Create convex/custom_agents/mutations.ts — follows the exact same pattern as all built-in agents:

export const chatWithCustomAgent = mutation({
  args: {
    customAgentId: v.id('customAgents'),
    threadId: v.string(),
    organizationId: v.string(),
    message: v.string(),
    maxSteps: v.optional(v.number()),
    attachments: v.optional(v.array(v.object({
      fileId: v.id('_storage'),
      fileName: v.string(),
      fileType: v.string(),
      fileSize: v.number(),
    }))),
  },
  handler: async (ctx, args) => {
    const authUser = await authComponent.getAuthUser(ctx);
    if (!authUser) throw new Error('Unauthenticated');

    // 1. Load custom agent from DB
    const agent = await ctx.db.get(args.customAgentId);
    if (!agent || !agent.isActive) throw new Error('Agent not found');

    // 2. Check team access (using shared helper)
    const userTeamIds = await getUserTeamIds(ctx, String(authUser._id));
    if (!hasTeamAccess(agent, userTeamIds)) {
      throw new Error('Agent not accessible');
    }

    // 3. Convert to SerializableAgentConfig
    const agentConfig = toSerializableConfig(agent);

    // 4. Resolve runtime config
    const { model, provider } = getDefaultAgentRuntimeConfig();

    // 5. Reuse chat hooks (RAG, attachments, etc.)
    const hooks = await createChatHookHandles(ctx);

    // 6. Compute knowledge scope — auto-narrow to agent's team
    const effectiveTeamIds = agent.teamId
      ? [`org_${args.organizationId}`, agent.teamId]
      : undefined; // null = use default (user's full teams)

    // 7. Delegate to existing pipeline
    return startAgentChat({
      ctx,
      agentType: 'custom',
      threadId: args.threadId,
      organizationId: args.organizationId,
      message: args.message,
      maxSteps: args.maxSteps,
      attachments: args.attachments,
      agentConfig,
      model: agentConfig.model ?? model,
      provider,
      debugTag: `[CustomAgent:${agent.name}]`,
      enableStreaming: true,
      hooks,
      userTeamIds: effectiveTeamIds, // Override — scoped to agent's team
    });
  },
});

3. Extend AgentType

Modify convex/lib/context_management/constants.ts:

export const AGENT_CONTEXT_CONFIGS = {
  // ... existing 6 agents unchanged ...
  custom: {
    modelContextLimit: DEFAULT_MODEL_CONTEXT_LIMIT,
    recentMessages: DEFAULT_RECENT_MESSAGES,
    outputReserve: 2048,
    enableSummarization: true,
  },
} as const;

4. CRUD Operations

convex/custom_agents/mutations.ts (additional):

• createCustomAgent — Create new agent config + initial version snapshot
• updateCustomAgent — Edit config + auto-create new version
• deleteCustomAgent — Soft delete (isActive: false)
• duplicateCustomAgent — Clone an agent
• rollbackCustomAgentVersion — Create new version from historical snapshot

convex/custom_agents/queries.ts:

• listCustomAgents — List active agents for org, filtered by hasTeamAccess()
• getCustomAgent — Get single agent by ID
• getCustomAgentVersions — Version history for an agent
• getAvailableTools — List tools with descriptions from TOOL_REGISTRY for picker UI

5. Required Change to startAgentChat

Add an optional userTeamIds parameter to StartAgentChatArgs:

// convex/lib/agent_chat/start_agent_chat.ts — small addition

export interface StartAgentChatArgs {
  // ... existing fields ...
  /** Override user team IDs (for custom agent team-scoped knowledge) */
  userTeamIds?: string[];
}

// In the function body, prefer the override when provided:
const teamIds = args.userTeamIds
  ?? (thread?.userId ? await getUserTeamIds(ctx, thread.userId) : []);
const userTeamIds = args.userTeamIds
  ? args.userTeamIds  // Already includes org prefix
  : [`org_${organizationId}`, ...teamIds];

This is backward-compatible — all existing agents omit the parameter and get the current behavior.

6. Router Integration (Optional Enhancement)

To let the routing agent automatically dispatch to custom agents:

• Dynamically inject custom agent descriptions into routing instructions
• Register custom agents as sub-agent tools (following the existing crm_assistant, web_assistant pattern)

---

Frontend Implementation

1. Global Team Filter

Add a team selector to the dashboard header (similar to the existing org context from URL):

• "All Teams" (default) → shows org-wide resources + resources from all user's teams
• Specific team → shows org-wide resources + that team's resources only
• Persisted in URL or local state (e.g., query param ?team=team-sales)
• Affects all team-scoped modules: custom agents, and future team-scoped resources
• Documents use the same hasTeamAccess() helper after migration

2. Agent Management Pages

List View (/settings/agents):

• Table of custom agents with name, team, status, tools count, last modified
• Create new / duplicate / delete actions
• Automatically filtered by global team filter

Detail/Edit View (/settings/agents/[id]):

• Basic Info: Name, display name, description, avatar
• Team: Dropdown to assign agent to a team (or org-wide)
• System Instructions: Monaco editor with syntax highlighting
• Tool Picker: Checkbox list grouped by category (data from TOOL_REGISTRY)
• Model Settings: Model preset selector, temperature slider, max tokens
• Knowledge: Toggle RAG, select top-k, link tone of voice
• Version History: Sidebar with rollback capability
• Preview/Test: Send test messages to see behavior

3. Agent Selector in Chat

• Add agent picker dropdown in chat interface (before sending first message)
• Show built-in agents: "Smart Router" (default), "CRM", "Documents", "Web", "Workflows", "Integrations"
• Show custom agents: filtered by global team filter + user's team membership
• Persist selected agent per conversation
• Allow switching agents mid-conversation (starts new context)

4. Tool Picker Component

Reusable component showing:

• Tool name and description (from TOOL_REGISTRY)
• Tool category (CRM, Knowledge, Documents, etc.)
• Checkbox for selection
• Usage guidance/warnings

---

Key Files to Modify

File	Changes
convex/schema.ts	Add customAgents, customAgentVersions tables
convex/lib/context_management/constants.ts	Add custom to AGENT_CONTEXT_CONFIGS
convex/agent_tools/tool_registry.ts	Export tool metadata for picker UI
convex/lib/agent_chat/start_agent_chat.ts	Add optional userTeamIds override parameter
convex/agents/chat/agent.ts	(Phase 4) Dynamic routing to custom agents
convex/documents/schema.ts	Add teamId + sharedWithTeamIds, later remove teamTags
convex/documents/get_documents_cursor.ts	Switch from teamTags to hasTeamAccess()
convex/lib/rls/helpers/rls_rules.ts	Update documents RLS to use new team fields
New: convex/lib/team_access.ts	Shared hasTeamAccess() helper (teamId + sharedWithTeamIds)
New: convex/custom_agents/config.ts	toSerializableConfig() conversion
New: convex/custom_agents/mutations.ts	CRUD + chatWithCustomAgent
New: convex/custom_agents/queries.ts	List, get, version history
New: app/routes/settings/agents/*	Management UI pages
New: components/agents/*	AgentPicker, ToolPicker, InstructionEditor
New: components/team-filter/*	Global team filter component

Not modified: generateAgentResponse(), createAgentConfig(), runAgentGeneration — these are reused as-is.

---

Phased Implementation

Phase 0: Team Fields Unification (prerequisite)

1. Create shared hasTeamAccess() helper in convex/lib/team_access.ts
2. Add teamId + sharedWithTeamIds fields to documents schema (alongside teamTags)
3. Query data distribution: count documents with 0, 1, and 2+ team tags
4. Run migration script to populate new fields from teamTags
5. Update get_documents_cursor.ts and rls_rules.ts to use hasTeamAccess()
6. Verify no regressions on documents
7. Remove teamTags field from documents schema

Phase 1: Data Model & Backend Core

1. Add customAgents and customAgentVersions tables to schema (using unified teamId + sharedWithTeamIds)
2. Add 'custom' to AGENT_CONTEXT_CONFIGS
3. Create toSerializableConfig() — DB record to SerializableAgentConfig
4. Add optional userTeamIds override to StartAgentChatArgs (backward-compatible)
5. Create chatWithCustomAgent mutation (reuses startAgentChat())
6. Implement CRUD mutations with auto-versioning
7. Implement version history queries and rollback

Phase 2: Agent Management UI

1. Agent list page (/settings/agents)
2. Agent create/edit form:
   • Basic info (name, description, avatar)
   • Team assignment dropdown
   • System instructions editor
   • Tool picker (checkboxes grouped by category, data from TOOL_REGISTRY)
   • Model preset selector
   • Knowledge settings
3. Version history sidebar with rollback

Phase 3: Chat Integration & Team Filter

1. Add global team filter to dashboard header
2. Add agent selector dropdown in chat interface
3. Show built-in agents + custom agents (filtered by team)
4. Wire selector to call chatWithCustomAgent mutation
5. Persist selected agent per conversation

Phase 4: Polish & Enhancements

1. Test/preview mode in agent editor
2. Duplicate agent functionality
3. Pre-built starter templates
4. Router integration — auto-dispatch to custom agents via routing agent
5. Workflow LLM node integration — allow LLM steps to reference a customAgentId to load agent config instead of manual prompt/tools configuration

---

Model Presets

Predefined presets mapped to existing environment variables and SerializableAgentConfig fields:

Preset	Env Variable	Use Case	SerializableAgentConfig Mapping
fast	OPENAI_FAST_MODEL	Quick responses, simple tasks	useFastModel: true
standard	OPENAI_MODEL	Balanced quality and speed	Default (no override)
advanced	OPENAI_CODING_MODEL	Complex reasoning, code generation	model: process.env.OPENAI_CODING_MODEL
vision	OPENAI_VISION_MODEL	Image analysis tasks	model: process.env.OPENAI_VISION_MODEL

UI: Radio buttons or dropdown with descriptions of each preset's strengths.

---

Security Considerations

1. Tool Access: Validate that selected tools exist in TOOL_REGISTRY
2. Prompt Injection: Sanitize user instructions (or accept risk with warning)
3. Model Access: Only allow presets the org has access to
4. Data Isolation: Team membership check before agent access; knowledge auto-scoped to agent's team — users can never access data from teams they don't belong to
5. Rate Limiting: Apply same rate limits as built-in agents (existing middleware)

---

Workflow Integration (Phase 4 Enhancement)

Custom agents can also be referenced within workflows by extending the LLM node:

• Add optional customAgentId field to LLMNodeConfig
• When set, load agent's systemInstructions + toolNames + modelPreset from DB
• Override the LLM step's manual configuration with the agent's saved config
• This lets workflow users select a pre-configured agent instead of manually writing prompts and picking tools per step

This reuses the custom agent definitions without modifying the workflow execution engine.

---

Verification Plan

1. Unit Tests: Test hasTeamAccess() with teamId only, teamId + sharedWithTeamIds, and org-wide resources
2. Unit Tests: Test toSerializableConfig() with various presets and configurations
3. Migration Tests: Verify documents migration preserves all access patterns:
   • teamTags: [] → teamId: null (org-wide access unchanged)
   • teamTags: ['sales'] → teamId: 'sales' (single-team access unchanged)
   • teamTags: ['sales', 'marketing'] → teamId: 'sales', sharedWithTeamIds: ['marketing'] (multi-team access unchanged)
4. Integration Tests: Create custom agent → chat with it → verify correct tools, instructions, and streaming
5. UI Tests: Verify CRUD operations in management UI
6. Manual Testing:
   • Create custom agent with subset of tools
   • Execute in chat and verify correct tool access + streaming
   • Test with different model presets
   • Verify team scoping:
     • Org-wide agent (teamId: null) visible to all members
     • Team agent (teamId: 'sales') visible only to Sales team members
     • Shared agent (teamId: 'sales', sharedWithTeamIds: ['marketing']) visible to both teams
     • Team agent's knowledge search is auto-narrowed to its team's data
     • Global team filter correctly shows/hides agents
   • Verify version history and rollback
   • Verify documents work correctly after migration (no regression)