Dolphin MCP Pilot: Operate DolphinScheduler via AI Agents

[Hui-of-limin]

Dolphin MCP Pilot: Operate DolphinScheduler via AI Agents

Hi DolphinScheduler community 👋

We've built Dolphin MCP Pilot, an MCP (Model Context Protocol) server that lets AI agents directly operate DolphinScheduler through natural language — from creating workflows and managing schedules to troubleshooting failures and performing backfills.

What is it?

An integration layer that exposes 58 DolphinScheduler operations as standardized MCP tools, enabling AI agents (Claude Desktop, Cursor, Cline, custom agents) to:

• Create SQL/DAG workflows with a single sentence
• Manage schedule lifecycle (create → online → offline → delete)
• Control process instances (pause / resume / rerun / rerun-from-failure)
• Intervene at task level (force success / skip failed nodes)
• Query logs and troubleshoot failures with guided next_action prompts
• Perform serial backfills with ordering guarantees (complementStartDate/complementEndDate)
• Manage resources and roll back workflow versions

Why this matters

Traditional DolphinScheduler interaction requires:

1. Opening the web UI
2. Navigating multiple pages
3. Manually clicking through operations
4. Checking logs and states separately

With Dolphin MCP Pilot, you can tell your AI agent:

"The ETL job failed yesterday. Why?"

The agent automatically:

1. Calls ds_list_process_instances → finds the failed instance
2. Calls ds_list_task_instances → locates the failed task node
3. Calls ds_get_task_log → pulls the error log
4. Reports the root cause

No manual clicking, no page-hopping.

Architecture

• Protocol: MCP over HTTP + SSE
• Authentication: Dual-mode (API Token or Username+Password)
• Compatibility: DolphinScheduler 3.x (tested on 3.2.1)
• Deployment: Docker / bare Python

Tool Coverage

Category	Tools	Examples
Project	5	ds_create_project, ds_list_projects
Workflow	14	ds_create_workflow, ds_update_task_param, ds_clone_workflow
Schedule	6	ds_set_schedule, ds_online_schedule, ds_complement_data
Instance	13	ds_rerun_process_instance, ds_force_task_success
Resource	10	ds_list_resources, ds_update_resource_content
Monitoring	6	ds_get_task_log, ds_list_task_instances
Raw API	4	ds_raw_get, ds_raw_post, ds_raw_put, ds_raw_delete

Quick Start

git clone https://github.com/iflytek/dolphin-mcp-pilot.git
cd dolphin-mcp-pilot

# Configure
cp .env.example .env
# Edit .env: set DS_URL, DS_TOKEN (or DS_USER/DS_PASSWORD)

# Run
docker compose --profile dev up -d

Connect your AI agent (example: Claude Desktop):

{
  "mcpServers": {
    "dolphinscheduler": {
      "url": "http://localhost:8001/mcp/",
      "headers": {
        "X-DS-Token": "your-token-here"
      }
    }
  }
}

Now you can talk to your agent:

"Create a daily SQL workflow in the data-team project that runs at 2 AM, querying SELECT * FROM user_behavior WHERE dt = '${bizdate}', with 3 retries on failure."

The agent orchestrates ds_create_workflow → ds_set_schedule → ds_online_schedule automatically.

Real-World Use Cases

Scenario 1: Automated troubleshooting

• Agent detects a failed workflow, traces the specific task node, pulls logs, and suggests fixes

Scenario 2: Bulk backfills

• "Rerun all daily reports from July 1 to July 31" → agent calls ds_complement_data with serial ordering

Scenario 3: Version control

• "Roll back user_analysis workflow to the previous version" → agent calls ds_rollback_workflow_version

Roadmap

• DolphinScheduler 2.x compatibility
• Pre-built DAG templates (CDC, feature engineering, report distribution)
• Multi-agent orchestration (scheduling agent ↔ data quality agent ↔ alerting agent)

Links

• GitHub: https://github.com/iflytek/dolphin-mcp-pilot
• License: Apache-2.0
• Docs: Installation · Configuration · Features · API Reference

---

We'd love to hear feedback from the community, especially on:

• API compatibility issues with different DS versions
• Authentication patterns in production environments
• Feature requests for operations not yet covered

This is an open-source project under Apache-2.0. Contributions, issues, and discussions are welcome!

[Hui-of-limin]

Quick update since posting this yesterday:

Ecosystem integration progress:

• Submitted to awesome-mcp-servers (the most comprehensive community list) - PR pending
• Submitted server.json + auto-publish workflow for MCP Registry (the official discovery path that replaced the modelcontextprotocol/servers README) - PR pending
• Now discoverable via GitHub search with 20 topics: mcp, model-context-protocol, apache-dolphinscheduler, claude, python, docker, scheduling, workflow-automation, data-engineering, ai-agent, etc.

One thing I'd love feedback on from the DS community:

The tool count is 58 right now, split across these categories:

Category	Count	Examples
Projects	5	create, list, rename, delete
Workflows	14	create DAG, update task params, clone, release, rollback version
Schedules	6	set cron, online, offline, complement (backfill)
Instances	13	pause, resume, rerun, rerun-from-failure, force success, skip
Resources	10	list, upload, download, update content, view, delete
Monitoring	6	task logs, instance lists with guided hints
Raw API	4	passthrough GET/POST/PUT/DELETE

Are there operations you find yourself doing repeatedly in the Web UI that aren't covered yet? I'm especially curious about cross-cutting workflows like "promote a workflow from dev to prod" or "diff two versions of a DAG definition" - the kinds of things that span multiple API calls and would save real time if an agent could do them in one shot.

GitHub: https://github.com/iflytek/dolphin-mcp-pilot
Docs: https://github.com/iflytek/dolphin-mcp-pilot/blob/main/README.md