feat: Integration package upload - allow custom REST API integrations

[larryro]

Summary

Allow users to create custom REST API integrations by uploading an integration package (zip file) containing config.json and connector.js.

Background

Currently, integrations are hardcoded in convex/predefined_integrations/. This limits users to only the pre-built integrations (Shopify, Circuly, Protel, Email).

Proposed Solution

Users upload a zip package with:

• config.json - metadata, operations definitions, secret bindings
• connector.js - JavaScript code that executes in sandboxed environment

Key Benefits

• Users can connect to any REST API
• No code changes needed to add new integrations
• Existing predefined integrations can be converted to packages (official examples)
• Foundation for future integration marketplace

Core Flow

1. User prepares integration package locally
2. Upload via UI
3. System validates config.json and connector.js
4. User fills in credentials (secrets)
5. Test connection
6. Save

Scope

• 4-step upload wizard UI
• Package parsing and validation
• Create integration from package (backend action)
• Official example packages for reference

Out of Scope (Phase 2+)

• TypeScript support (can add later with esbuild)
• Migration of predefined integrations to packages
• Integration marketplace

[larryro]

Detailed Implementation Plan

Integration Package Structure

my-integration.zip
├── config.json       # Metadata + operations definitions
└── connector.js      # Connector code (JavaScript)

Note: Phase 1 supports JavaScript only. TypeScript support can be added later using esbuild transpilation.

config.json Example

{
  "name": "my-api",
  "title": "My Custom API",
  "description": "Connect to My Custom API",
  "version": 1,

  "authMethod": "api_key",
  "secretBindings": ["apiKey", "baseUrl"],
  "allowedHosts": ["api.example.com"],

  "operations": [
    {
      "name": "list_users",
      "title": "List Users",
      "description": "Get all users from the API",
      "operationType": "read"
    },
    {
      "name": "create_user",
      "title": "Create User",
      "operationType": "write",
      "requiresApproval": true
    }
  ],

  "connectionConfig": {
    "timeout": 30000
  }
}

connector.js Example

const connector = {
  execute: function(ctx) {
    const { operation, params, http, secrets } = ctx;

    const baseUrl = secrets.get('baseUrl');
    const apiKey = secrets.get('apiKey');

    const headers = {
      'Authorization': 'Bearer ' + apiKey,
      'Content-Type': 'application/json'
    };

    if (operation === 'list_users') {
      const response = http.get(baseUrl + '/users', { headers });
      if (response.status !== 200) {
        throw new Error('API error: ' + response.text());
      }
      return { success: true, data: response.json() };
    }

    // ... other operations
  }
};

---

Connector Code API

API	Description
ctx.operation	Current operation name (string)
ctx.params	Operation parameters (object)
ctx.secrets.get(name)	Get secret value
ctx.http.get(url, { headers })	GET request
ctx.http.post(url, body, { headers })	POST request
ctx.base64Encode(str)	Base64 encode
ctx.base64Decode(str)	Base64 decode

---

New Files

services/platform/app/features/settings/integrations/components/
  integration-upload/
    integration-upload-dialog.tsx         # Upload dialog
    steps/
      upload-step.tsx                      # Step 1: Upload zip
      preview-step.tsx                     # Step 2: Preview config
      credentials-step.tsx                 # Step 3: Fill credentials
      test-step.tsx                        # Step 4: Test connection
    utils/
      parse-integration-package.ts         # Parse zip package
      validate-config.ts                   # Validate config.json
    hooks/
      use-upload-integration.ts            # Upload state management

services/platform/convex/integrations/actions/
  create_from_package.ts                   # Create integration from package

services/platform/public/integration-packages/  # Official example packages
  shopify.zip
  example-rest-api.zip

---

Files to Modify

1. UI Entry Point

File: app/features/settings/integrations/components/integrations-client.tsx

• Add "Add Integration" button (upload entry)
• Render IntegrationUploadDialog

2. Translations

File: messages/en.json

Add upload-related translation keys.

---

Wizard Steps

Step 1: Upload Zip

• Drag & drop upload area
• File size limit (1MB)
• Parse and validate package content
• Provide official example package download links

Step 2: Preview Config

• Integration name, description
• Operations list
• Secret bindings list
• Connector code preview (read-only, syntax highlighting)

Step 3: Fill Credentials

Dynamically generate form based on secretBindings:

• One input for each secret
• Password type for sensitive fields

Step 4: Test Connection

• Select a read operation to test
• Fill test parameters
• Execute and display result

---

config.json Schema

interface IntegrationConfig {
  // Required
  name: string;           // Identifier
  title: string;          // Display name
  secretBindings: string[];  // Required secrets list
  operations: Array<{
    name: string;
    title?: string;
    description?: string;
    operationType?: 'read' | 'write';
    requiresApproval?: boolean;
  }>;

  // Optional
  description?: string;
  version?: number;
  authMethod?: 'api_key' | 'bearer_token' | 'basic_auth';
  allowedHosts?: string[];
  connectionConfig?: {
    timeout?: number;
    rateLimit?: number;
  };
}

---

Official Integration Packages

Convert existing predefined integrations to zip packages in /public/integration-packages/:

1. shopify.zip - Shopify Admin API
2. example-rest-api.zip - Simple example (using JSONPlaceholder)

Users can download these as reference or upload directly.

Migration Path (Optional Future Work)

• Phase 1: Add upload functionality, keep predefined integrations
• Phase 2: Convert predefined integrations to official packages
• Phase 3: Remove convex/predefined_integrations/ code

---

Verification Plan

1. Package Validation Tests:

   • Valid/invalid zip files
   • Missing required files
   • config.json format errors

2. Creation Tests:

   • Upload example package to create integration
   • Verify integration data stored correctly

3. Execution Tests:

   • Call newly created integration via Agent
   • Verify operations execute correctly

---

Key Reference Files

• integrations-client.tsx - UI entry point
• shopify.ts - Existing connector code reference
• execute_integration_internal.ts - Execution environment
• create.ts - Create action reference
• schema.ts - Data model