Repository navigation
First-class durable subscription sets for scheduled Workflow fan-out #12387
frost-leo
started this conversation in
Temporal Backend
Replies: 1 comment
|
I’m not quite sure what you’re requesting. Do you want a new Schedule action type that starts multiple workflows at each scheduled time, with the ability to add or remove workflows between runs? |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Is your feature request related to a problem? Please describe.
Temporal Schedules provide a durable model for when to start a Workflow, and
Workflow Executions provide a durable model for how to run it. A recurring
application pattern is still left to every user to build: running the same
scheduled Workflow over a large, mutable set of subscribed entities while
freezing the exact membership used by each occurrence.
Examples include polling devices, refreshing tenant data, recurring billing,
inventory synchronization, monitoring targets, campaign audiences, and data
collection. Items are added, paused, resumed, or removed while the Schedule is
active.
Applications generally choose one of these approaches:
Workflow query, snapshot, paginate, fan out, deduplicate, and reconcile it.
The third approach is often the most practical, but it creates a dual
consistency boundary between Temporal and the external subscription store.
Every application must independently implement membership revisions, snapshot
isolation, occurrence identity, duplicate suppression, pagination, crash
recovery, reconciliation, and visibility.
A concrete example is a daily Workflow over 10,000 collection targets. The set
may change while the Schedule fires, but every occurrence must use one exact,
explainable snapshot. Pagination must not skip or duplicate items because the
external set changed during planning, and a process crash must not create a
second logical execution of the same item for that occurrence.
Describe the solution you'd like
Please consider a first-class durable subscription set (or versioned target
set) that can be bound to a Schedule action. The important part is the
semantics rather than a specific API:
add, update, pause, resume, and remove operations.
mutations affect either that occurrence or a future one, never an ambiguous
partial set.
child Workflow IDs can be derived deterministically.
copied into one Workflow input or History event.
does not silently omit or duplicate members.
through Temporal APIs and, eventually, the UI.
data reference; Temporal does not need to understand business data.
One possible execution model is:
This is not a request for exactly-once effects in external systems. It is a
request for one durable consistency model covering Schedule occurrence creation
and the target-set revision selected by that occurrence.
If a server primitive is considered too broad, an official cross-SDK
abstraction or reference pattern with explicit snapshot, pagination,
Continue-As-New, deterministic identity, and recovery semantics would still be
valuable.
Describe alternatives you've considered
cardinality and poor support for bulk lifecycle changes and batch execution.
not always appropriate for a periodically changing batch population.
design History rollover, membership revisioning, paging, and visibility.
custom dual-write/idempotency protocol and reconciliation between the
external store and Temporal.
Additional context
This pattern applies to billing populations, IoT fleets, monitoring targets,
tenant synchronization, inventory refresh, campaigns, and data pipelines. A
Temporal-owned primitive or canonical abstraction could prevent these systems
from repeatedly building incompatible subscription control planes around the
service.
Related proposals with different scopes:
Event History events and notification triggers.
registration and synchronization of Schedules.
Neither appears to cover versioned target-set membership frozen per Schedule
occurrence.
I can provide a concrete application use case and help clarify the expected
semantics and failure scenarios, but I am not currently able to implement this
server-level feature.
All reactions