Proposal: Unify Macro & Template

[yangzhang75]

Summary

A Macro is a reusable, parameterized piece of workflow. The "Template" idea
from #6070 is really just an edge case of a Macro — a macro used to generate
a whole workflow. So instead of two separate features, we treat Macro as the
general concept.

A Macro can be used two ways:

1. Inside a workflow — embed it as a building-block node.

2. To generate a workflow — fill its parameters to produce a runnable
   workflow. This is the real use case for biologists.

Background

#6070 proposed Templates: on a whole workflow, mark some operator properties
as configurable, let a user fill a small form, and generate a new runnable
workflow. We also have Macros (#5115): reusable sub-graphs you drop into a
workflow. These are the same underlying idea — so rather than maintain two
features, a Template becomes a special case of a Macro.

The two ways to use a Macro

1. Inside a workflow (building block). Drop a Macro node onto the canvas;
   macros can be nested.
2. To generate a workflow. A dedicated page shows a
   read-only preview plus a form of just the macro's configurable parameters;
   fill them and Run to produce an independent, runnable workflow. This is what
   Proposal: Workflow Templates — save a workflow as a reusable, parameterized template #6070 called a Template.

How Macro and Template relate

• A macro that isn't runnable on its own → a building block used inside a
  workflow (way 1).
• A macro that is runnable on its own → can generate a complete workflow
  (way 2). That's the old Template.

Sharing

A Macro can be shared / made public and browsed in the Hub, like workflows.

Status

A working prototype exists. Design details are still being worked out — this is
mainly to align on the overall direction.

Continues #6070.

[chenlica]

@yangzhang75 @mengw15 and I had several offline discussions and came up with this proposal.

[yangzhang75]

Feedback from testing

• File parameter should defaults empty — each user picks their own file
• Auto-layout / fit / zoom controls in the macro preview
• A run keeps running in the background and stays findable after leaving the page

Decisions so far (open to discussion)

• Cloning first: clone a public macro from the Hub, then generate
• The workflow preview can be hidden for a simpler, biologist-friendly interface
• Parameter form cleaned up: hide internal keywords so biologists only fill what they need

[Neilk1021]

Could a Macro carry its own documentation? A macro-level description plus optional per-parameter help text which is rendered directly in the parameter form?

This seems especially valuable for the workflow-generation case, where the person filling out the form may not know what the macro does or what a given parameter expects. Letting the author supply that context inline means the guidance travels with the macro rather than living in a separate doc or getting lost. That would make public macros in the Hub considerably more usable, and it fits well with the "hide internal keywords" direction already mentioned.

[yangzhang75]

Thanks for raising this, it's the direction we're going. I'm working on letting the macro author edit both the macro-level description and per-parameter help text so the context travels with the macro.

[mengw15]

On "To generate a workflow": where should Run live?

Currently the macro page has a "Run workflow" button, but clicking it actually creates a workflow first under the hood. This becomes confusing once the user leaves the page and comes back:

• If the macro is a general building block (not itself a workflow), the executions launched from this page arguably should not be listed on this page at all.
• Either way, UX-wise the user may no longer be able to find the execution they started — it belongs to a workflow they never explicitly saw.

Two possible designs:

Design 1 — a macro is a workflow. A cloned macro is a workflow; clicking Run runs it right here and never creates a new workflow. To reuse the same macro with different parameters, you clone it again.

Design 2 — a macro is where workflows get created. One macro can create multiple workflows with different parameters. In this design I do not think the macro page should run anything directly: the button should be "Create workflow", which jumps to the generated workflow's canvas, and the user clicks Run there.

I lean toward Design 2. With Design 1, macros and workflows sit in separate tabs yet both are effectively "workflows" — that is confusing and not a good design language. Design 2 adds a small learning curve, but all the user needs to learn is to click the Run button on the canvas; the macro author can also put a tutorial in the macro description (which can be markdown). It also resolves the "a run keeps running in the background and stays findable" feedback above for free — the execution lives with the created workflow, not with the macro page. And it stays consistent with what was agreed in #6070: each generation produces a new, normal workflow the user owns.

[mengw15]

One possible extension of Design 2: maintain the macro → workflow creation relationship, so that opening a macro also shows all the workflows created from it, together with their execution status (this is essentially open question 3 in #6070). That would solve findability inside the macro page itself. But I think this design gets very complex — versioning, what exactly to display, etc. — and may end up giving users a new kind of burden instead. I'd rather accept the small learning curve: Texera users do not need to build workflows by themselves, but they should understand the concept of a workflow.

[carloea2]

I think this unification starts from the wrong premise.

A Macro is a workflow: it contains operators and their connections. Whether that graph runs independently or is embedded inside another workflow changes how it is used, not what it is. In my opinion, a macro should not be distinguishable from a workflow therefore no need for macro concept.

A Template is not a graph. It is a reusable set of property values that can be applied to compatible workflows.

For example, suppose two completely different workflows both contain a CSV Reader. A template could specify:

CSVReader.file = "control-patients.csv"

That template can configure both workflows without knowing, or caring, how their DAGs are connected. The workflows can be rearranged, extended, or have operators removed. The template remains an independent configuration; properties without a matching operator are simply not applied.

I would also like templates to be selectable directly from the workflow canvas. This would let users switch the same workflow between different contexts without changing its structure.

For example, a biological-analysis workflow could offer:

• Control Cohort Template

  • Input file: control-patients.csv
  • Significance threshold: 0.05
  • Output: control-results

• Treatment Cohort Template

  • Input file: treatment-patients.csv
  • Significance threshold: 0.01
  • Output: treatment-results

Selecting a different template on the canvas would update those properties while leaving the DAG untouched. The same templates could also be applied to other compatible workflows.

The distinction is simple:

• Macros (workflows) reuse computation and structure.
• Templates reuse configuration and context.

These concepts are orthogonal, not special cases of each other. A template could be applied to a regular workflow or to a macro. Treating a template as a runnable macro unnecessarily ties configuration to one particular DAG and removes the valuable ability to reuse and switch configurations across different workflows.

TL;DR:
Configured workflow=Workflow graph+Template properties