Skip to content

Extension models need built-in support for idempotent create operations #618

@stack72

Description

@stack72

Problem

When a workflow partially fails and is re-run, model create methods execute again for resources that were already successfully created. This leads to duplicate resources for non-idempotent APIs (e.g. Droplets, Load Balancers) because the auto-generated models are simple POST wrappers with no state awareness.

Real-world scenario encountered:

  1. Workflow with VPC, Tag, Droplets, LB, and Firewall
  2. VPC creation fails (invalid CIDR), Tag succeeds
  3. User fixes the CIDR and re-runs the workflow
  4. Tag step runs again (harmless — DO tags are idempotent)
  5. If all steps had succeeded initially, a re-run would create duplicate droplets and load balancers

The context.dataRepository is available for models to check existing state, but every model author has to implement this pattern themselves. The auto-generated DigitalOcean extension models don't do this at all — their create methods unconditionally POST to the API.

Proposed Solution

Provide a framework-level mechanism so model authors get idempotent creates without boilerplate. Some options:

  1. createOrGet helper in context — A utility that checks existing data before calling create, e.g. context.createOrGet("state", instanceName, () => create(...)). If state data already exists, it returns the existing handle; otherwise it runs the create function and stores the result.

  2. Declarative idempotent: true on methods — swamp skips execution if the method's output spec already has data for this model instance. Simple and zero-effort for model authors.

  3. Workflow-level "skip if data exists" on steps — Let workflow authors mark steps that should be skipped when the target model already has data, e.g. skipIfExists: true.

  4. Code generation improvement — The auto-generated models should check context.dataRepository for existing state before creating, and optionally verify the resource still exists via a GET call.

Scope of Changes

  • Model execution context — Add helper methods or pre-execution checks
  • Extension model code generation — Generated create methods should include state checks
  • Workflow engine (if option 3) — Add step-level skip conditions based on data existence
  • Documentation — Best practices for idempotent model design

The core issue is that partial workflow failures are inevitable (API errors, rate limits, transient failures), and re-running is the natural recovery path. Without idempotency, re-running is risky rather than safe.

Metadata

Metadata

Assignees

Labels

betaIssues required to close out before public betadocumentationImprovements or additions to documentationenhancementNew feature or request

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions