Proposal: Workflow Templates — save a workflow as a reusable, parameterized template

[yangzhang75]

What it does

In current Texera main, sharing a workflow means sharing the whole editable graph. If someone wants to reuse a colleague's pipeline with their own input file or a different threshold, they have to open the full canvas, find the right operator, and edit it by hand — easy to get wrong, and it exposes the whole pipeline to accidental edits.

This proposal adds Workflow Templates. An author turns a workflow into a template and marks a few operator properties as configurable. Anyone with access then instantiates their own workflow by filling in a short form for just those properties — without touching, or even seeing, the rest of the graph. The result is a normal, runnable workflow they own.

The case that motivated it: a lab publishes an analysis pipeline (e.g. a scGPT single-cell workflow), and end users just point it at their own dataset and hit run.

flowchart TD
    A["Author's workflow"] -->|"Make Template"| B["Opens in template mode"]
    B -->|"tick the configurable properties,<br/>then save"| C["Template<br/>(stored and shared like a workflow)"]
    C -->|"open → POST /build<br/>(idempotent: created once)"| D["Instantiated workflow<br/>(the user owns it)"]
    D --> E["Build page<br/>form on top, runnable preview below"]
    E -->|"Submit → POST /update<br/>(whitelisted keys only, new version)"| D
    E -->|"Run the preview"| F["Results"]

Loading

How it works

Authoring. From any workflow, the author clicks Make Template and checks which operator properties should stay editable. The workflow is saved as a template in which everything is locked except those checked properties (stored per operator as a configurableProperties whitelist). It's stored and shared like a workflow — owner plus per-user access.

Instantiating. Anyone with access builds their own workflow from the template. They can change only the checked properties; the rest of the graph is fixed and not editable. Values are applied server-side into the workflow content, restricted to the whitelist, and stored as raw JSON so typed values (like file references) round-trip losslessly.

A new Parameter source operator lets a template expose typed inputs (key/value pairs, file references) as first-class graph inputs.

The build page

The "Build workflow from template" page shows the configurable-property form on top, and a live, read-only but runnable preview of the workflow below it — so a user can build, run, and see results on one page, without being able to edit the graph.

Open questions

1. One template → one workflow, or one → many? Today a template maps to a single instantiated workflow (1-to-1), so any template change naturally flows to that workflow. Should we instead allow one template to produce multiple named instances — e.g. the same template run against different datasets, each saved separately?
2. Should a template-built workflow be distinguished from a normal one? It's meant to be run and re-parameterized, not freely edited. Is that distinction worth surfacing (a label, a separate section), or should it just look like any other workflow?
3. Should template-built workflows appear on the main Workflows page at all, or live only under Templates?
4. When an admin disables the feature, what happens to workflows already built from templates? The feature can be turned off at runtime from the admin settings (it hides the Templates tab and the "create template" button). Should the workflows those templates already produced be hidden too, or stay visible as normal, runnable workflows the user owns?

Feedback welcome on the overall approach and on these four.

[SarahAsad23]

1. I think it makes sense for a single template to map to multiple workflows. As you mentioned, users may want to reuse the same template with different parameters, datasets, or configurations while keeping each instantiated workflow separate.
2. It would be helpful to mark workflows that were created from a template. That makes it easier for users to distinguish between workflows they built from scratch and those generated from templates, which helps with organization and tracking.
3. I think it's best to keep template-built workflows under the Templates page rather than mixing them into the main Workflows page. Otherwise, it could become confusing and clutter the main workflow list.
4. From an implementation standpoint, it would be simpler to hide template-built workflows when the admin disables the feature. However, from a user perspective, I think it would be better if existing workflows remained visible and runnable since users already own them and may still rely on them.

[yangzhang75]

@SarahAsad23 quick clarification, if a single template produces multiple workflows (point 1), and template-built workflows live under the Templates page (point 3), how do you see the Templates page organizing them? For example, would each template show its N derived workflows nested under it?

[chenlica]

This "template" feature is based on the requests from many users, especially in the medical domain. In particular, biologists and "Web Lab researchers" don't want to use a workflow as it is too "complicated" to them. They prefer a simpler "form" interface on which they can just change a few parameters.

Regarding the four questions:

1. We can allow a template to produce multiple workflows. A question is how to manage the "linkage" between a template and its workflow. I suggest once a workflow is generated, we can just keep the information about the corresponding template. The user can still modify the template, and we allow the template and the workflow to be inconsistent.

2. For each workflow created by a template, we just keep the ID of the template. Other than that, the workflow is the same other "normal workflows."

3. We show templates and workflows separately. We can show a special tag for a template-generated workflow.

4. This "template" feature can be turned on or off before Texera is deployed. We don't allow the admin to turn it or off after the deployment.

[yangzhang75]

Thanks for the context. One more thing I want to discuss that I didn't cover in the original questions, the preview workflow under the Templates page. My current design keeps it read-only (users can run but not edit; any modification requires generating a linked workflow first, and the linked workflow on the Workflows page becomes fully editable like any other workflow). Does that match how you'd expect it to work?

Also, I'm wondering if we should have a duplication check (for both templates and linked workflows). If a user builds an exact copy of a workflow from a template with the same parameter values, should we block or warn them, or just let it happen?

[mengw15]

1. Before answering the four questions, I think we should first be clear about what a template is. In my understanding, a template is a tool that helps users create workflows. Like printing: a plate exists to produce many copies, so a template should be able to produce multiple workflows — that is the semantically correct model. I think 1-to-1 is acceptable as an MVP product, but on the roadmap 1-to-many is a must.

2. Second, a workflow created from a template is no different from a normal workflow — the only difference is how it was created. Since there is no difference in what it is, the way we treat workflows should not change because of this feature. As for the case where some users don't want to click into a workflow just to run it — that is orthogonal to templates. We can address it later with options like a "run workflow / export result" button directly on the workflow dashboard. (Following the same logic, I'd lean toward letting the owner freely edit a template-generated workflow like any other, rather than keeping it locked to the whitelisted parameters.)

3. Third, template-generated workflows should appear on the main Workflows page. I agree with @chenlica that we can add a special tag to distinguish them.

4. Fourth, once a workflow has been generated from a template, it is a workflow — so it should always remain visible, regardless of whether this feature is turned on or off. To use another example: today a user can create a workflow either by uploading a workflow JSON or by building it manually. Whether or not the JSON-upload feature continues to exist, I believe those workflows should always continue to exist.

[yangzhang75]

That framing makes sense to me.

A few follow-up thoughts:

1. Preview vs linked workflow: with the lock gone, my current split is that the preview under the Templates page stays read-only (users can run but not edit; any modification means generating a linked workflow first), and the linked workflow on the Workflows page is fully editable like any other workflow. Does that match how you'd expect it to work?

2. Template page UX: since the linked workflows live on the Workflows page, would it help to add a button on the Templates page that jumps directly to the user's linked workflow (already editable), so users don't have to switch pages to find it?

3. Parameter versioning: after a user edits parameters in their linked workflow, should the template's default parameters update to the latest submitted version (so future users start from the most recent state), or stay locked to the values the template author originally set? Updating feels easier to keep in sync, but keeping the original feels safer against accidental drift.

[chenlica]

@mengw15 Regarding your first point about "what a template is," its main motivation is to make it easier for some users (e.g., biologists) who don't like a workflow interface to easily create a workflow. A template can be viewed as "a workflow with placeholders or parameters." A template is NOT designed to create multiple workflows.

@yangzhang75 : In this discussion, let's ignore the current implementation. We should focus on the user experience, then decide how to implement it.

In general, a key question is: after a workflow is created, if we change the template, do we need to update the correspond workflow? My preference is "No," so that the maintenance becomes easier.

[mengw15]

Agreed on the motivation — the goal is to give users who find the canvas too complicated a simple, form-based way to create a workflow.

On "a template is NOT designed to create multiple workflows": understood, but I think purpose and cardinality are orthogonal — "mass production is not the goal" doesn't mean the design should assume exactly one workflow per template.

In fact, I think your preference on the key question settles the cardinality question. The rationale for 1-to-1 was that "any template change naturally flows to that workflow" — i.e., 1-to-1 exists to make propagation easy. If we prefer no propagation (which I agree with: easier maintenance, and no surprises for the workflow owner), then that rationale goes away, and nothing else in the design requires 1-to-1. Meanwhile sharing already produces multiple instances naturally: a template shared with N users gives each of them their own workflow, and even a single user may run the same pipeline on two datasets and keep both. So the clean model is exactly what you suggested earlier: the generated workflow just stores the template ID, workflows evolve independently after creation, and no structural assumption of 1-to-1 anywhere.

[chenlica]

Thanks. I think we are on the same page.

If we agree on this user experience, @yangzhang75 can discuss the design.

@Yicong-Huang : This discussion so far focuses on the user experience. As a general practice, should the design remain in the same discussion or in a new discussion?

[Yicong-Huang]

In general, it's better to start a new discussion, so each discussion has one topic. This is easier to manage. make sure to refer to each other.

[Yicong-Huang]

For this particular case, I see we agreed on UX. and looks like we agreed to introduce this new UX. So the discussion phase is completed.

Next we need a todo item (issue) that summarizes the agreed UX. @yangzhang75 please open an issue referring this discussion.

Then code implementation can be discussed on the corresponding PR whiling looking at the code itself.

[Yicong-Huang]

Hmm, I feel this is similar to macro idea. Suppose I have A->B ->C workflow (each as an operator).

• I can select B -> C to create a template, I will get a configurable template. I can reuse this template with the rest of the workflow.
• If I instead select B-> C and create a macro, I will get a Marco op with its own configurable properties, which are mapped to the underneath operators. I can also reuse this Marco operator with the rest of the dag.

The only difference is template is represented as a subDAG, and macro is represented as a high-level operator with the subDAG details hidden. I feel we could unify them (e.g., give user an option to expand a macro to see the subDAG, or collapse a subDAG (template) into a Marco.

[chenlica]

@Yicong-Huang I think you misunderstood the meaning of a "template." Consider a workflow A -> B -> C, where B is a filter operator with a condition "price < 20". If we want to make "20" a configurable variable, we can create a template that has X as a parameter. After a user specifies a value for X, say, 50, the system will generate a workflow where B has a condition "price < 50."

I hope you can see it's very different from a macro.

[yangzhang75]

Following the UX we've converged on above, here's the concrete design to implement it.

Agreed decisions

1. Cardinality — 1‑to‑n. A template can produce many workflows — sharing it with N users, or running it on different datasets, naturally yields multiple separate instances.
2. A generated workflow is just a normal workflow — user‑owned, fully editable, like any other; it records which template it came from (for the tag). The only difference is how it was created. (How tightly it stays tied to that template afterward is open — see Open question 4.)
3. A "from template" tag distinguishes generated workflows.
4. Always visible regardless of the feature toggle — once generated, it is a workflow (like one created by uploading a JSON), so it persists whether the feature is on or off.
5. The feature toggle is set before deployment, not at runtime.
6. Configurable properties = a per‑operator configurableProperties whitelist stored in the workflow content; the server applies submitted values only for keys in that whitelist (e.g. a Filter's price < 20 becomes price < X; the user sets X, and the server refuses to write any non‑whitelisted property).

Motivation: give users who find the canvas too complex (e.g. biologists) a simple, form‑based way to create a workflow.

Data model

template holds the template (a snapshot of the workflow content, incl. each operator's configurableProperties whitelist). workflow_of_template maps each generated workflow back to its origin template — one template → many workflows (tid not unique), used only for the "from template" tag / provenance.

erDiagram
    template ||--o{ workflow_of_template : "1 template -> N workflows"
    workflow ||--o| workflow_of_template : "origin of each generated workflow"
    template {
      serial tid PK
      varchar name
      varchar description
      text content "graph + per-operator configurableProperties whitelist"
      timestamp creation_time
      timestamp last_modified_time
    }
    workflow_of_template {
      int wid PK "FK -> workflow, ON DELETE CASCADE"
      int tid FK "-> template, nullable"
      timestamp created_at
    }
    workflow {
      int wid PK
      varchar name
      text content
    }

Loading

Ownership/sharing reuse template_of_user(uid, tid) and template_user_access(uid, tid, privilege), same as workflows.

User flow

• Authoring: from any workflow, the author clicks Make Template and ticks which operator properties stay editable (stored per operator as the configurableProperties whitelist). Saved and shared like a workflow.
• Using: anyone with access opens the template, fills the configurable form (with an optional read‑only preview — see Open question 1), and hits Submit → a new, fully‑editable workflow they own, marked with a "from template" tag. Each Submit creates a new workflow.

Services

Single /api gateway → distinct services: amber (/template/*, /templated-workflow/build + /{wid}/update, /workflow/*, execution WebSocket); workflow-compiling-service (/compile, preview schema propagation + validation); file-service (/dataset, file parameters); computing-unit-managing-service (/computing-unit/*, run unit); PostgreSQL via DAO/jOOQ.

Open questions

1. Inline runnable preview on the build page. Keep the read‑only‑but‑runnable preview under the form — so after changing a parameter and clicking Submit, the user sees the resulting workflow right below and can run it immediately (without leaving the page)? Or drop it and just create the workflow?

2. Template deletion. When a template is deleted, should its generated workflows keep showing "from template" (keep the id), or lose the tag (tid set null)?

3. A "generated from this template" list on the template page. Should the template page list the workflows generated from it (with, e.g., generation time and version history), so users can find them?

4. Coupling after generation. We agreed each Submit creates a new workflow. What tie does a generated workflow keep to its template? Options:

   • live link — the user can later re‑apply the template's form to update that same workflow;
   • fully detached — keep only the id for the tag; manual editing only from then on;
   • template keeps a runnable snapshot — the template retains the first generated version, so the user can run it directly on the template page.

   Hard case for the "live link" option: if the user deletes the operators the template marked configurable, the template can no longer update that workflow (nothing left to write).