Skip to content

ComfyUI HTTP API vs MCP

Jump to bottom
Benji edited this page Aug 23, 2026 · 2 revisions

ComfyUI in Calliope: HTTP API vs MCP

This page is the design record for how Calliope talks to ComfyUI. It is the source for the GitHub wiki.

Status: current as of v1.2.0. Every product generate uses the HTTP job queue. The Comfy MCP bridge (comfy_mcp plugin) is shipped but not loaded — the agent must not see comfy_run_workflow / comfy_run_template / comfy_search_templates / other MCP comfy_* tools. The only comfy_* tool in the live harness is comfy_server_info, a native HTTP health check (ComfyUIClient.health()), not an MCP call.

What Calliope owns

Calliope is a local story → assets → video studio. ComfyUI is the render engine. Calliope never reads or writes Comfy’s input/ or output/ folders. Settings only need ComfyUI Base URL (and Dry-run). Folder paths stay on the ComfyUI side.

Imported workflows live in Calliope’s SQLite workflows table (API-format JSON, role tags, prompt profile, enabled flag). Jobs live in Calliope’s queue. Generated files land under Calliope assets_dir.

Project → Assets has three types: Characters, Environments (locations), and Misc. Items (items). All three can be generated as reference images. Only characters and locations feed video reference slots — items do not.

Two ways to reach ComfyUI

Calliope has two Comfy surfaces. They are not interchangeable. Only the HTTP queue is live.

Calliope HTTP queue (canonical, live) Comfy MCP (comfy_* tools, not loaded)
Identity SQLite workflow_id (Settings → Workflows) Filesystem workflow_path (or MCP templates)
Client ComfyUIClient in calliope-backend/src/calliope/comfyui/client.py comfy-mcp stdio subprocess (COMFYUI_URL = the same base URL)
How a run starts Insert a jobs row → worker MCP run_workflow / generate_image / templates
Job list Project Queue, Playground Queue, SSE job.created, artifact cards MCP’s own job object
Pause / retry / poll timeout Settings → Queue (queue_poll_timeout_sec, default 1800s, 0 = forever). Agent wait_for_jobs uses the same default. MCP’s wait/watch
Prompt fill Role tags (Input:prompt) via smart_fill_inputs Whatever the MCP tool/file expects
HITL requires_approval on enqueue_asset_jobs / enqueue_video_jobs / run_workflow Would need the same guard if the plugin is reloaded
When to use Assets, Script/Video generate, Playground, Agent @workflow Ad-hoc Comfy exploration if the plugin is loaded again
Agent visibility list_workflows, run_workflow, enqueue_*, comfy_server_info None — comfy_mcp is not in _PLUGIN_MODULES 
flowchart TB
  subgraph calliope [Calliope]
    UI[Assets / Playground / Agent]
    Q[jobs table]
    W[queue worker]
    Lib[workflows table]
  end
  subgraph http [ComfyUI HTTP]
    Up["POST /upload/image"]
    Prompt["POST /prompt"]
    Hist["GET /history/{id}"]
    View["GET /view"]
  end
  subgraph mcp [Comfy MCP — shipped, not loaded]
    Unused[comfy_mcp plugin]
    Stdio[comfy-mcp stdio]
  end

  UI --> Q
  Lib --> W
  Q --> W
  W --> Up --> Prompt --> Hist --> View
  Unused -.-> Stdio
  Stdio -.-> Prompt
Loading

The solid path is the product. The dotted path exists in source only — do not re-enable it for @workflow generates.

Canonical path: HTTP API + job queue

Every product generate (Assets, per-scene Video, Playground, Agent run_workflow) uses the same worker.

  1. Fill by role, not node id. Calliope discovers editable nodes from _meta.title tags in API Format JSON (Display Name (Input:prompt), (Input:width), (Output:video), …). smart_fill_inputs writes prompt / refs / duration onto those roles. See the README workflow section.
  2. Enqueue. queue_manager.enqueue(project_id, kind, workflow_id, payload={input_values, source}). source is "playground" or "agent" etc. The job is visible in Queue immediately.
  3. Worker (calliope-backend/src/calliope/queue/worker.py):
    • Load workflow_json for workflow_id
    • patch_workflow(workflow, input_values)
    • prepare_media_inputs: if a LoadImage / LoadAudio / LoadVideo input looks like a local path, POST {base}/upload/image (Comfy uses the image form field for audio and video too)
    • POST {base}/prompt with { prompt: patched_json, client_id }
    • Poll GET {base}/history/{prompt_id} until done, failed, or queue_poll_timeout_sec
    • Download via GET {base}/view → save under assets_dir/{project_id}/{kind}/
    • Attach outputs to the character / location / scene when the payload says so
  4. Dry-run (Settings, default off) skips Comfy and writes a placeholder PNG/MP4. Unreachable Comfy fails the job — it does not silently fake images.

Worker uploads image, audio, and video files to Comfy. H3 Motion Context clip indexes are filled from the timeline ((Input:clipindex) primitives); the worker only rewrites the per-project latent folder (calliope/p{id}/). First/Next motion pairs and per-scene Chain from prev clip still go through this same queue.

Who enqueues

Entry Tool / route Project
Assets Generate backend asset enqueue current project
Video Generate clip backend video enqueue current project
Playground Generate POST /api/playground/generate hidden playground scratch project
Agent @workflow harness run_workflow linked project, or the hidden Playground scratch in a sandbox chat

run_workflow is Playground generate with Calliope library identity: same smart_fill_inputs, same input_values, same worker. It is requires_project=False and requires_approval=True.

  • Sandbox (project_id NULL): jobs land on the hidden Playground scratch. The agent must not create_project just to generate an image. File a finished picture onto a film later with attach_asset (character_sheet / location / item).
  • Linked: jobs land on that project’s Queue. create_project / link_project only when the user wants a film, not because @workflow was tagged.

enqueue_asset_jobs / enqueue_video_jobs still need a linked project (characters, locations, scenes). wait_for_jobs works in both modes; pass job_ids from the enqueue result (required in sandbox).

Agent @workflow mentions

The Agent composer stays a chat box, not a second Playground form.

  • @ typeahead lists enabled Calliope workflows (GET /api/workflows).
  • A chip serializes as @Name in visible prose plus a structured mention { type: "workflow", id, name, kind }.
  • Attachments upload through POST /api/playground/uploads (allowlisted, under assets_dir/uploads).
  • Persist: mentions / attachments on the user/message event.
  • LLM history gets a short appendix, e.g.
[Calliope context]
workflow_id=12 name="krea2-t2i" kind=image
attached: …/uploads/ab12-ref.png (image)

HITL regexes (generate / image / video) still read the user’s own words, not this appendix (otherwise kind=image would auto-approve renders). Text edits (add_item, add_character, generate_story, …) are not render permission.

The model is instructed: a workflow_id= appendix means call run_workflow, not list_workflows guess, and not any MCP comfy_* generate tool. Aspect ratio in prose (16:9) maps to the same 1920×1080 / 1280×720 presets Omni already uses. Call comfy_server_info first; if Comfy is unreachable or dry-run is on, stop.

Why this version does not use ComfyUI MCP

The bridge file exists (calliope-backend/src/calliope/agent/harness/plugins/comfy_mcp.py) but is not listed in plugins/__init__.py::_PLUGIN_MODULES. MCP comfy_* tools fail against a typical local install and burn the agent step budget. @krea2-t2i must not go through MCP even if the plugin is reloaded later.

1. Different identity. @ names a Calliope library row (workflow_id). MCP comfy_run_workflow wants a JSON file path. Bridging them means dumping workflow_json to a temp file on every mention. That dump is a new source of truth that can drift from Settings (prompt profile, enabled flag, role tags).

2. Queue is the product contract. Assets, Playground, Video, Queue pause, retries, poll timeout, wait_for_jobs, and chat artifact cards all assume a jobs row. MCP bypasses that table. A tagged “generate with my imported workflow” would disappear from Queue and from project history.

3. Fill logic already exists on the HTTP path. MiniMax H3 six-section rewrite, ordered (Input:image) slots, (Input:character) / (Input:location), duration from the scene, First/Next motion context — all of that is smart_fill_inputs + prompt profiles + the worker. MCP would not get it unless we reimplemented it against a temp file.

4. HITL + session mode stay one policy. run_workflow is requires_approval=True and works in sandbox or a linked project. Dumping to MCP would need a second approval story and a place to put outputs.

5. MCP is the wrong tool for the mention. MCP would be useful when the agent is talking to Comfy itself: node/model search, validate a file, launch/stop Comfy, run a workflow that was never imported. @ is the user pointing at a workflow they already imported into Calliope. Live health is already comfy_server_info over HTTP.

So: MCP stays disabled. @ → Calliope id → run_workflow → queue. Re-add comfy_mcp to _PLUGIN_MODULES and call register() from build_harness only when the comfy-mcp server is a supported path.

What MCP would be for (if re-enabled)

Do not prefer comfy_* generate tools today — they are not in the tool list. If the plugin is loaded again, it would be for talking to Comfy itself, not for a Calliope workflow_id:

  • comfy_system_stats / node and model search / comfy_validate_workflow — inspect the live Comfy install
  • comfy_launch_comfyui / comfy_stop_comfyui — process lifecycle
  • comfy_run_workflow — a JSON path or an MCP template, not a Settings → Workflows id

Do not translate Calliope workflow_id into a temp file “just to use MCP.” Health before a render is already comfy_server_info (HTTP).

Out of scope / later

  • Dumping Calliope JSON to disk so @ can call comfy_run_workflow
  • @ mentions for characters, locations, items, or jobs
  • Mounting OmniComposer on /agents (Omni binds nodeId → input_values for one workflow; the Agent composer binds chat context)
  • Attaching Misc. Items to scene / video reference slots (items generate on Assets only)

If a later version wants MCP for tagged workflows, the missing piece is an explicit export: write a Calliope workflow to a known path, run MCP, then import the outputs back into the jobs table. Until that exists, the HTTP queue remains the only complete path.

Code map

Piece Where
HTTP client + health calliope-backend/src/calliope/comfyui/client.py (comfy_server_info)
Role tags / smart fill calliope-backend/src/calliope/comfyui/roles.py, smart_fill.py, parser.py
Patch + worker calliope-backend/src/calliope/comfyui/patcher.py, queue/worker.py
Playground generate / uploads / scratch project calliope-backend/src/calliope/routers/playground.py
Agent run_workflow / enqueue / attach_asset calliope-backend/src/calliope/agent/harness/plugins/render.py
Harness load list (no MCP) calliope-backend/src/calliope/agent/harness/plugins/__init__.py
Comfy MCP bridge (not loaded) calliope-backend/src/calliope/agent/harness/plugins/comfy_mcp.py
@ composer calliope-web/src/lib/components/agent/AgentComposer.svelte
Public workflow contract README — ComfyUI workflows

Clone this wiki locally