Complete Verified Guide: Self-Host AI BYOK on 0.27.x (5 undocumented requirements + model matrix + troubleshooting)

[ryo9999]

AFFiNE 0.27.x Self-Host AI (BYOK) — Complete Verified Setup Guide

A complete, battle-tested walkthrough for getting AI BYOK + full-text search working on a stock 0.27.4 Docker deployment. Everything below was verified end-to-end (probes, chat, doc reads, hybrid search) on a real deployment. The official docs cover some of this, but four undocumented requirements and one model-capability matrix are what actually make it work.

TL;DR — the 5 things the docs don't tell you

1. Env AFFINE_BACKEND_RUNTIME_CONFIG_PATH=/root/.affine/config/config.json — the Rust native runtime (which actually runs the AI) does not read ~/.affine/config/config.json by default (Self-host: AI chat always fails with copilot_disabled — the native runtime never reads config.json #15506). Without it, GraphQL says Copilot is enabled but every chat returns copilot_disabled.
2. Env DEPLOYMENT_TYPE=selfhosted — without it the runtime assumes Cloud and customEndpointMode becomes unavailable, hiding the custom-endpoint option in the BYOK dialog ([Bug]: AI BYOK cant set custom ai providers in self-hosted version #15534).
3. server.externalUrl must equal the URL you actually browse from — the CORS allowlist is built from it exactly. Mismatch = every BYOK probe shows "Connection failed" while keys/URLs/models are all fine ([Bug]: [Self-host BYOK] Three undocumented failure modes: CORS externalUrl mismatch, 16-token structured probe, Peer disconnected on long action turns #15552).
4. Full-text (lexical) search requires the Manticore container per the official indexer guide — env vars AFFINE_INDEXER_ENABLED=true + AFFINE_INDEXER_SEARCH_ENDPOINT, NOT config.json. The embedded provider in config.example.json is canary-only.
5. BYOK keys are configured in Workspace Settings → Integrations → AI BYOK (not the admin panel — the old admin AI settings were removed in 0.27.x).

Final working architecture

browser ─ https://your-host:3010 (tailscale serve / reverse proxy)
  └ AFFiNE 0.27.4 (affine_server)
       ├─ BYOK → new-api / OpenRouter (OpenAI-compatible)
       │    ├─ z-ai/glm-5.3-flash      → Chat & writing, Actions, Image understanding
       │    ├─ qwen/qwen3-embedding-8b → Workspace indexing
       ├─ BYOK → OpenRouter
       │    ├─ google/gemma-4-31b-it   → Structured output
       │    └─ openai/gpt-4.1-nano     → Search reranking (real logprobs)
       ├─ Indexer → manticoresearch/manticore:10.1.0 (lexical search)
       ├─ PostgreSQL pgvector (vector search + DB)
       └─ Redis

config/config.json

{
  "$schema": "https://github.com/toeverything/affine/releases/latest/download/config.schema.json",
  "server": {
    "name": "AFFiNE Self-hosted",
    "externalUrl": "https://your-public-url:3010"
  },
  "copilot": {
    "enabled": true,
    "byok": {
      "enabled": true,
      "allowedProviders": ["openai", "anthropic", "gemini", "fal"],
      "allowCustomEndpoint": true,
      "allowPrivateEndpoint": true
    }
  }
}

(allowPrivateEndpoint: true is required when your gateway is on a private network — e.g. a tailnet hostname.)

docker-compose services (the full working set)

services:
  affine:
    image: ghcr.io/toeverything/affine:stable
    ports: ['127.0.0.1:3010:3010']
    volumes:
      - ./data/storage:/root/.affine/storage
      - ./config:/root/.affine/config
    environment:
      - REDIS_SERVER_HOST=redis
      - DATABASE_URL=postgresql://affine@postgres:5432/affine
      - AFFINE_BACKEND_RUNTIME_CONFIG_PATH=/root/.affine/config/config.json
      - DEPLOYMENT_TYPE=selfhosted
      - AFFINE_INDEXER_ENABLED=true
      - AFFINE_INDEXER_SEARCH_ENDPOINT=http://indexer:9308
    depends_on:
      redis: {condition: service_healthy}
      postgres: {condition: service_healthy}
      affine_migration: {condition: service_completed_successfully}
      indexer: {condition: service_healthy}

  affine_migration:
    image: ghcr.io/toeverything/affine:stable
    volumes:
      - ./data/storage:/root/.affine/storage
      - ./config:/root/.affine/config
    command: ['sh', '-c', 'node ./scripts/self-host-predeploy.js']
    environment:
      - REDIS_SERVER_HOST=redis
      - DATABASE_URL=postgresql://affine@postgres:5432/affine
      - AFFINE_BACKEND_RUNTIME_CONFIG_PATH=/root/.affine/config/config.json
      - AFFINE_INDEXER_ENABLED=true
      - AFFINE_INDEXER_SEARCH_ENDPOINT=http://indexer:9308
    depends_on:
      postgres: {condition: service_healthy}
      redis: {condition: service_healthy}
      indexer: {condition: service_healthy}

  indexer:
    image: manticoresearch/manticore:10.1.0
    ports: ['127.0.0.1:9308:9308']
    volumes: ['./manticore:/var/lib/manticore']
    ulimits:
      nproc: 65535
      nofile: {soft: 65535, hard: 65535}
      memlock: {soft: -1, hard: -1}
    healthcheck:
      test: ['CMD', 'wget', '-O-', 'http://127.0.0.1:9308']
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis
    healthcheck:
      test: ['CMD', 'redis-cli', '--raw', 'incr', 'ping']
      interval: 10s

  postgres:
    image: pgvector/pgvector:pg16
    volumes: ['./data/postgres:/var/lib/postgresql/data']
    environment:
      POSTGRES_USER: affine
      POSTGRES_DB: affine
      POSTGRES_INITDB_ARGS: '--data-checksums'
      POSTGRES_HOST_AUTH_METHOD: trust
    healthcheck:
      test: ['CMD', 'pg_isready', '-U', 'affine', '-d', 'affine']
      interval: 10s

Start: docker compose up -d. Expose: tailscale serve --bg --https=3010 http://127.0.0.1:3010 or your reverse proxy.

Configuring BYOK keys

1. Open a workspace → Settings → Integrations → AI BYOK → Add key
2. Provider OpenAI → storage Server → paste API key
3. Enable "Use a custom API-compatible endpoint" (only visible when DEPLOYMENT_TYPE=selfhosted is set — see trap Feat/style #2)
4. Endpoint: https://your-gateway/v1, API dialect: Chat Completions API
5. Add models manually with their use-cases checked
6. Test connection, then Connect. ⚠️ One failed model probe blocks saving the whole profile ([Bug]: [v0.27.4] Workspace BYOK profiles cannot be saved when recommended-model probes partially fail #15496) — uncheck any failing model and save.

Model selection — what actually works (probe request captured & verified)

The probes are much stricter than they look:

• Structured-output probe sends max_completion_tokens: 16 with strict json_schema and "Reply with OK." → reasoning models (glm, deepseek, qwen-flash) fail invalid_response because reasoning eats the entire 16-token budget. Use a small non-reasoning model for structured output (e.g. google/gemma-4-31b-it).
• BYOK "rerank" is NOT a /rerank endpoint. It sends chat/completions with logprobs: true, top_logprobs: 5 and scores p(yes)/(p(yes)+p(no)). Consequences:
  • dedicated rerankers (qwen3-reranker-8b) 500 on this route → probe unavailable
  • gemini/mistral/llama accept logprobs but return null → probe passes but score is always 0.0 (decorative only)
  • only OpenAI-family models return real logprobs → openai/gpt-4.1-nano ($0.10/$0.40 per 1M) is the cheapest actually functional reranker (verified: relevant doc scores 1.0, unrelated 0.0)
• Anthropic BYOK is broken on 0.27.4 (/v1/v1/messages doubled → 404, BYOK: Anthropic provider always fails — request URL contains /v1 twice (404) #15505). Use an OpenAI-compatible gateway instead.
• Verified model matrix:

Use case	Model	Probe
Chat & writing / Actions / Image understanding	z-ai/glm-5.3-flash	✅
Workspace indexing	qwen/qwen3-embedding-8b	✅
Structured output	google/gemma-4-31b-it	✅ (3/3)
Search reranking	openai/gpt-4.1-nano	✅ (real scores)
Image generation / Transcription	— needs image-output / audio-input capability; leave unset if unused	—

Troubleshooting quick reference

Symptom	Cause	Fix
chat returns copilot_disabled	runtime didn't read config.json	env AFFINE_BACKEND_RUNTIME_CONFIG_PATH (#15506)
custom endpoint option missing	runtime thinks it's Cloud	env DEPLOYMENT_TYPE=selfhosted (#15534)
every probe "Connection failed"	CORS: browsing from a different origin than externalUrl	set externalUrl = actual URL, look for Blocked CORS request from origin in logs (#15552)
LEXICAL_UNAVAILABLE in search results	manticore container missing	indexer service per official guide; notice is benign (vector-only fallback)
structured probe invalid_response	reasoning model + 16-token budget	use a non-reasoning model for structured
rerank unavailable	dedicated reranker vs ChatLogprobs protocol	use an OpenAI-family chat model
doc-save action fails Peer disconnected	upstream drops stream on huge long-session payloads	retry in a fresh session; not a config error (#15552 Defect 3)

Version notes

• Verified on the official 0.27.4 stable image. The embedded search provider in repo config.example.json does not exist in the stable image — use Manticore per the indexer guide.
• Related issues: BYOK: Anthropic provider always fails — request URL contains /v1 twice (404) #15505 (anthropic), Self-host: AI chat always fails with copilot_disabled — the native runtime never reads config.json #15506 (config read), [Bug]: AI BYOK - unable to configure, connection failed #15530/[Bug]: AI BYOK cant set custom ai providers in self-hosted version #15534 (connection failed / custom endpoint), [Bug]: [Self-host BYOK] Three undocumented failure modes: CORS externalUrl mismatch, 16-token structured probe, Peer disconnected on long action turns #15552 (CORS + probe details), [Bug]: [v0.27.4] Workspace BYOK profiles cannot be saved when recommended-model probes partially fail #15496 (one failed probe blocks save).