Workflow Engine

Workflows (tapestries) are MuxCore's killer feature. They're multi-step automation pipelines that coordinate many modules to accomplish a goal — like acquiring a movie from request to notification — without hardcoding any step.

⚠️ Contract-defined, not yet implemented. The WorkflowEngine interface is defined in pkg/contracts/workflow.go, but no runtime implementation ships with core. The Scheduler contract handles cron-style tasks today. Workflow orchestration requires a module that implements the WorkflowEngine contract.

What Is a Tapestry?

In the *arr world, workflows are implicit and hardcoded:

Sonarr: Search → Download → Move → Rename → Notify

You can't change the steps. You can't add steps. If you want subtitles fetched before import, that's a separate program (Bazarr) with its own scheduler.

In MuxCore, workflows are explicit, configurable, and extensible:

You define: Request → Metadata → Search → Download → Verify → Extract → Analyze → Transcode → Subtitles → Import → Notify

Each step is handled by a different module. The workflow engine orchestrates — it doesn't implement any step directly.

A Real Example

Here's a tapestry for importing a movie:

id: movie-request
name: "Movie Request Pipeline"
steps:
  - name: metadata-lookup
    handler: metadata-tmdb
    retry: 3
    timeout: 30

  - name: indexer-search
    handler: indexer-prowlarr
    retry: 2
    timeout: 60
    depends_on: [metadata-lookup]

  - name: download
    handler: downloader-qbittorrent
    retry: 1
    timeout: 0       # no timeout — downloads can take hours
    depends_on: [indexer-search]

  - name: verify
    handler: verifier-builtin
    retry: 3
    timeout: 300
    depends_on: [download]

  - name: extract
    handler: extractor-unpackerr
    retry: 2
    timeout: 600
    depends_on: [verify]

  - name: analyze
    handler: analyzer-builtin
    retry: 1
    timeout: 120
    depends_on: [extract]

  - name: transcode
    handler: transcoder-ffmpeg
    retry: 1
    timeout: 3600
    depends_on: [analyze]

  - name: subtitles
    handler: content-subtitle
    retry: 2
    timeout: 120
    depends_on: [transcode]

  - name: library-import
    handler: media-movies
    retry: 1
    timeout: 60
    depends_on: [subtitles]

  - name: notify
    handler: notifier-discord
    retry: 2
    timeout: 10
    depends_on: [library-import]

How It Works

Starting a Tapestry

// Discover the workflow engine via gRPC
disc := discoveryv1.NewDiscoveryServiceClient(conn)
resp, _ := disc.FindByCapability(ctx, &discoveryv1.FindByCapabilityRequest{
    Capability: "workflow.engine",
})
// Use the discovered engine module
engine := workflowv1.NewWorkflowEngineClient(conn)
runID, err := engine.Run(ctx, &workflowv1.RunRequest{
    DefinitionName: "movie-request",
    Params: map[string]string{
        "title":   "The Terminator",
        "quality": "2160p",
    },
})

The Engine Executes Each Step

Step 1: metadata-tmdb → looks up movie details
    ↓ success
Step 2: indexer-prowlarr → searches for releases
    ↓ success  
Step 3: downloader-qbittorrent → starts download, waits for completion
    ↓ success
Step 4: verifier-builtin → checks file integrity
    ↓ success
Step 5: extractor-unpackerr → unpacks archives
    ↓ success
Step 6: analyzer-builtin → checks codec, resolution, bitrate
    ↓ success
Step 7: transcoder-ffmpeg → creates optimized versions
    ↓ success
Step 8: content-subtitle → fetches subtitles
    ↓ success
Step 9: media-movies → imports into library
    ↓ success
Step 10: notifier-discord → sends completion message

If a Step Fails

Step 4: verifier-builtin → FAILS (corrupt download)
    ↓
Retry 1: wait 5 seconds → FAILS
    ↓
Retry 2: wait 25 seconds → FAILS
    ↓
Retry 3: wait 125 seconds → FAILS
    ↓
Tapestry fails → notification sent → manual review

Key Features

DAG Support

Steps declare dependencies with depends_on. Steps with no dependencies (or whose dependencies are all satisfied) run in parallel:

metadata-lookup ─┬─→ indexer-search-1 (torrent)
                 └─→ indexer-search-2 (usenet)
                          │
                    first one wins → download

Retries with Exponential Backoff

When a step fails, the engine retries with increasing delays:

Attempt 1: immediate
Attempt 2: 5 second delay
Attempt 3: 25 second delay
Attempt 4: 125 second delay

Each step configures its own retry count. Backoff is configurable via RetryPolicy (initial delay, max delay, factor, jitter).

Idempotency

Running the same request twice doesn't cause double downloads or double imports. The engine integrates with IdempotencyProvider:

idempotencyKey := hash(mediaType + title + quality + season + episode)

If that key was already processed, the engine returns the cached result instead of re-executing.

Timeouts

Each step has its own timeout. Downloads can be unlimited (timeout: 0). Metadata lookups might have 30 seconds. Transcodes might have an hour.

Step-to-Step Data Passing

Each step produces output (StepResult.Output), and subsequent steps can reference it via TapestryStep.InputMapping:

- name: transcode
  handler: transcoder-ffmpeg
  input_mapping:
    source_file: "$steps.download.output.filepath"
    target_quality: "$params.quality"

Planned Built-In Tapestries

MuxCore will ship with default workflow definitions. Users can modify these or create their own:

Tapestry	Purpose
`movie-request`	Full movie acquisition — search through import
`tv-request`	TV episode acquisition
`music-request`	Music album acquisition
`book-request`	Book acquisition
`media-transcode`	Re-encode existing media
`media-migrate`	Move media between storage providers
`media-backup`	Backup to cold storage
`library-scan`	Scan and import existing media files

Note: These tapestries are designed templates — they do not ship with the current build. They will be included when the WorkflowEngine implementation module is available. In practice, you'll customize them for your setup — different downloaders, different notification services, optional steps.

Advanced Features (Planned)

Compensation — Undo on Failure

If step 7 fails after step 3 succeeded, the engine runs compensation handlers:

Step 3: download → started        [compensation: cancel download]
Step 7: import → started          [compensation: remove partial import]
Step 8: notify → not reached

Conditional Steps

Skip steps based on results:

- name: transcode
  handler: transcoder-ffmpeg
  condition: "media.codec != 'h264'"   # only transcode if not already h264

Fallback Chains

Try handlers in order until one succeeds:

- name: metadata-lookup
  handlers: [metadata-tmdb, metadata-tvdb, metadata-imdb]

Power Use Cases

Intelligent Orchestration

Move workload to idle GPU — transcoding scheduler picks the least-loaded worker
Auto-balance storage — move media to the provider with the most free space
Prioritize by popularity — recently watched media stays on fast storage
Predictive pre-transcoding — transcode the next episode while you're watching the current one

Cross-Media Awareness

A tapestry can coordinate across media types — something impossible in siloed *arr stacks:

User adds an anime series
  → Checks for manga adaptation
  → Checks for light novel source
  → Offers to track and download all related media

Workflow Engine Interface

// Defined in pkg/contracts/workflow.go
type WorkflowEngine interface {
    RegisterDefinition(ctx context.Context, def TapestryDefinition) error
    RemoveDefinition(ctx context.Context, name string) error
    GetDefinition(ctx context.Context, name string) (*TapestryDefinition, error)
    ListDefinitions(ctx context.Context) ([]TapestryDefinition, error)
    Run(ctx context.Context, name string, params map[string]any) (string, error)
    Status(ctx context.Context, runID string) (*TapestryRun, error)
    Cancel(ctx context.Context, runID string) error
    Pause(ctx context.Context, runID string) error
    Resume(ctx context.Context, runID string) error
    ListRuns(ctx context.Context, filter TapestryRunFilter) ([]TapestryRun, error)
}

Key types:

TapestryDefinition — the workflow template (name, steps, metadata)
TapestryStep — a single step with handler, retry, timeout, depends_on, input_mapping
TapestryRun — a running instance with status, step results, start/end times
StepResult — output of a completed step, passed to dependent steps
TapestryRunFilter — filter by status, definition name, time range

Discovered via FindByCapability("workflow.engine"). If no engine module is registered, the Scheduler contract handles cron-style tasks independently.

Next Steps

Module System — modules that handle workflow steps
Event System — how step transitions are communicated
Writing Modules — build a module that participates in workflows

Uh oh!

Workflow Engine

Workflow Engine

What Is a Tapestry?

A Real Example

How It Works

Starting a Tapestry

The Engine Executes Each Step

If a Step Fails

Key Features

DAG Support

Retries with Exponential Backoff

Idempotency

Timeouts

Step-to-Step Data Passing

Planned Built-In Tapestries

Advanced Features (Planned)

Compensation — Undo on Failure

Conditional Steps

Fallback Chains

Power Use Cases

Intelligent Orchestration

Cross-Media Awareness

Workflow Engine Interface

Next Steps

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Wiki Navigation

Clone this wiki locally