Definition of Workflow

[karthik18495]

I think this is a good idea, And just for my clarification and understand better the nomenclature, and also i think, Wen may also be interested in this conversation

Defining the Workflow in utilities is a good ides. Just a note here is that We have MultiStepJobs defined in scheduler_epic. Which basically can run multi-step jobs as DAG. And each job stage can be defined as parallel jobs as well. So we can make use of this and move the job and its related definitions to workflows (aid2e/utilities/workflows/*).

1. Workflow will consists of a set of layers:
   • For each layer, there will be a scheduler attached right? The scheduler currently takes care of submitting the job step, monitoring it and reporting when it is done. So the base class of scheduler has this.
   • However, there can be multiple jobs defined in the same layer. For instance in the Q2 example, when simulating, we can think of it as generating N parallel simulations, So for this step we will have N Layers?
2. Handling multiple independent objectives:
   • Consider the Holistic optimization. where each objective is basically a workflow. and each workflow can be run in parallel and involves multiple steps. Hence, I may I propose having to call each design point evaluation as a Workflow and each Workflow can have N branches?

Do you think this definition (nomenclature ) for the workflow works?

• Workflow: an end-to-end evaluation unit (e.g., one design point evaluation).
• Branch: an optional subgraph inside a workflow (useful when you want multiple independent pipelines under one workflow, e.g., “physics sim” branch + “surrogate” branch). You can also skip this and model everything as one DAG; branches just make it easier to group steps.
• Layer / Stage: a logical step group where multiple jobs can run in parallel (fan-out) and then later stages depend on their completion (fan-in).
• Job / Task: the smallest schedulable unit (one simulation, one training run, one postprocess, etc.).
• Scheduler: the runtime executor for jobs (submit, monitor, collect status/artifacts).

And the core model for this workflow and its related objects can be

WorkflowDefinition
 ├─ id / name / metadata
 ├─ branches: List[BranchDefinition]              # optional; can default to single branch
 └─ build() -> WorkflowInstance

BranchDefinition
 ├─ id / name
 └─ dag: DagDefinition (nodes=StageDefinition or JobDefinition)

StageDefinition
 ├─ id / name
 ├─ scheduler: SchedulerSpec                      # what scheduler to use for jobs in this stage
 ├─ job_factory: JobFactory                       # creates 1..N jobs based on context
 ├─ parallelism: ParallelismPolicy                # limits, chunking, retries, etc.
 └─ outputs: ArtifactSpec

JobDefinition (optional if you always generate via Stage)
 ├─ id / name
 ├─ scheduler: SchedulerSpec
 ├─ command / payload / resources
 └─ outputs: ArtifactSpec