feat(parallelism): make sub-application ID an overridable hook (#761)#781
Open
elijahbenizzy wants to merge 3 commits into
Open
feat(parallelism): make sub-application ID an overridable hook (#761)#781elijahbenizzy wants to merge 3 commits into
elijahbenizzy wants to merge 3 commits into
Conversation
Previously, the sub-application id for parallel sub-apps was computed inline inside MapActionsAndStates._create_task as a deterministic hash of (parent_app_id, key). That hash is what enables resume-on-rebuild for parallel sub-apps (retry-on-failure, crash recovery), but it also means that with a cascading state initializer the sub-apps will replay stale state on every parent invocation -- the case reported in #761. The previous attempt (#778, closed) tried to auto-fix this by salting sub-app ids with context.sequence_id. That broke test_end_to_end_parallel_collatz_many_unreliable_tasks, because sequence_id advances on parent rebuilds and the retry path could no longer find the previously-persisted sub-app checkpoints. This change keeps the default behavior unchanged (deterministic (parent_app_id, key) hash, so retry-on-failure still works) and instead exposes the id computation as a named, overridable method on TaskBasedParallelAction. Users who hit #761 override the hook to add whatever salt they need; everyone else is unaffected. Adds one test exercising the new hook with a cascading initializer.
skrawcz
approved these changes
May 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an overridable
sub_application_idhook onTaskBasedParallelActionfor users who want sub-app cache semantics different from the parent's (e.g. fresh-each-invocation under a cascading state initializer).Background
TaskBasedParallelAction(and itsMapStates/MapActions/MapActionsAndStatessubclasses) compute each sub-application id as a deterministic hash of(parent_app_id, key). That's load-bearing for retry-on-failure and crash recovery: when the parent is rebuilt (e.g. inside a retry loop, or viainitialize_from), each sub-task gets the same id it had on the prior attempt, so a cascading state initializer can find its persisted checkpoint and resume. The retry-on-failure path intests/test_end_to_end.py::test_end_to_end_parallel_collatz_many_unreliable_tasksdepends on this.This default is consistent with
initialize_from(..., resume_at_next_action=True): you opted into cascading checkpoint resume, and you get it on sub-apps too. That's by design. But users sometimes want different cache semantics for sub-apps -- e.g. fresh execution per parent invocation, or pinning to a business key. Today there's no clean way to express that without overridingtasks()and reaching intotask.application_idpost-hoc.What this PR does
Promotes the inline sub-app id computation into a named, overridable method:
TaskBasedParallelAction.sub_application_id(key, state, context). The default implementation is unchanged -- it still returns_stable_app_id_hash(parent_app_id, key)-- so existing behavior is preserved bit-for-bit. Users who want different sub-app cache behavior override the hook on their subclass.Example
```python
import uuid
from burr.core import ApplicationContext, State
from burr.core.parallelism import MapStates
class MyParallel(MapStates):
# ... states(), action(), reduce(), reads, writes ...
```
Background context
PR #778 was an earlier attempt that changed the default to salt with
context.sequence_id-- that broke the retry-on-failure path becausesequence_idadvances on every parent rebuild. That PR was closed after analysis. See #778 for the diagnostic discussion. Conclusion: keep the default, surface the choice to the user.#761 is the user report that surfaced this. Per the analysis there, the reported behavior is by design (cascading initializer + deterministic ids = checkpoint resume on sub-apps too). This PR adds the escape hatch.
Test plan
tests/core/test_parallelism.pypass (19/19) -- default hook returns the same value the inline computation did.tests/test_end_to_end.py::test_end_to_end_parallel_collatz_many_unreliable_taskspasses -- the default is preserved, so the retry path is unaffected.tests/core/test_parallelism.py::test_sub_application_id_override_enables_fresh_execution_with_cascading_initializer-- overrides the hook to demonstrate fresh-execution-per-invocation, asserts all 9 sub-task executions run rather than 3 running and 6 replaying.