Skip to content

RFC: Run full workflows via CLI with --workflow #259

@FernandoCelmer

Description

@FernandoCelmer

Summary

Today dotflow start --step module.func runs a single @action step. To execute a workflow composed of multiple task.add(...) calls you have to wrap everything inside an @action-decorated function that calls .start() itself — which mixes step and orchestrator semantics and makes the CLI unable to control the execution mode.

This RFC proposes a new, explicit entry point: --workflow.

Motivation

Current pain — projects like dotflow-cloud end up with:

@action
def main():
    workflow = DotFlow(config=Config())
    workflow.task.add(step=step_one)
    workflow.task.add(step=step_two)
    workflow.start(mode="sequential")
    return workflow

Problems:

  • @action semantically means one step; here it hides a whole
    orchestrator
  • .start() is hardcoded inside; the CLI can't override mode,
    storage, etc.
  • --step resolves this as a step and would execute it as one task,
    not as a pipeline

Proposal

Add --workflow (or -w) to the start command. It resolves a factory module.attr that returns a DotFlow instance, and the CLI is responsible for starting it.

Usage

dotflow start --workflow dotflow_cloud.workflow.main
dotflow start --workflow dotflow_cloud.workflow.main --mode parallel

Factory shape

from dotflow import Config, DotFlow
from dotflow_cloud.actions import step_one, step_two

def main() -> DotFlow:
    workflow = DotFlow(config=Config())
    workflow.task.add(step=step_one)
    workflow.task.add(step=step_two)
    return workflow  # no @action, no .start() here

CLI behavior

  • --step and --workflow become mutually exclusive (one required)
  • With --workflow: import factory → call it → assert it returned a DotFlow → call workflow.start(mode=self.params.mode)
  • --mode, --storage, --path continue to work (CLI wins over factory defaults when explicitly passed)

Proposed changes

  1. dotflow/cli/setup.py — add --workflow in a mutually exclusive group with --step
  2. dotflow/cli/commands/start.py — branch on params.workflow, new private _start_from_factory() using Module(self.params.workflow)
  3. dotflow/core/exception.py — new InvalidWorkflowFactory raised when the resolved attr is not callable or does not return DotFlow
  4. Docs + example in docs_src/ showing the factory pattern

Open questions

  1. Override semantics — should CLI flags (--mode, --storage) always override factory config, or only when the factory did not set them? Recommendation: CLI always wins
  2. Dotted attribute supportModule already supports pkg.module.attr via rpartition, so no change there
  3. Schedule parity — should dotflow schedule --workflow ... get the same treatment?
  4. Return-type validation — should we allow factories that return None and call .start() themselves (backwards compat), or strictly require a returned DotFlow?

Non-goals

  • No change to @action semantics
  • No deprecation of --step — it remains the way to run a single task

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions