Performance [P2]: Reduce orchestration history replay allocations

[berndverst]

What is the issue?

Long orchestration histories are copied and scanned multiple times before and during replay.

What is the impact?

Replay CPU time and allocation pressure grow with history length. This can raise worker latency and reduce throughput for long-running orchestrations, particularly when many instances are replayed concurrently.

Details about the issue including code reference

Relevant code:

durabletask-python/durabletask/worker.py

Lines 1135 to 1143 in 55d8e0b

	# Extract parent trace context from executionStarted event
	parent_trace_ctx = None
	orchestration_name = "<unknown>"
	for e in list(req.pastEvents) + list(req.newEvents):
	if e.HasField("executionStarted"):
	orchestration_name = e.executionStarted.name
	if e.executionStarted.HasField("parentTraceContext"):
	parent_trace_ctx = e.executionStarted.parentTraceContext
	break

durabletask-python/durabletask/worker.py

Lines 2135 to 2187 in 55d8e0b

	orchestration_name = "<unknown>"
	orchestration_started_events = [e for e in old_events if e.HasField("executionStarted")]
	if len(orchestration_started_events) >= 1:
	orchestration_name = orchestration_started_events[0].executionStarted.name
	self._orchestration_name = orchestration_name
	self._logger.debug(
	f"{instance_id}: Beginning replay for orchestrator {orchestration_name}..."
	)
	if not new_events:
	raise task.OrchestrationStateError(
	"The new history event list must have at least one event in it."
	)
	# Check for rewind BEFORE replay. A rewind is indicated by an
	# executionRewound event in new_events. We look for an
	# executionCompleted event in the committed history (old_events)
	# to decide whether to rewind or replay:
	# 1. executionCompleted IS present → the orchestration reached a
	# terminal state (e.g. failed). This is a *new* rewind that
	# the worker must short-circuit by building clean history.
	# 2. executionCompleted is NOT present → the backend already
	# processed the RewindOrchestrationAction and removed
	# executionCompleted from the committed history. Here the
	# executionRewound event in new_events acts as a "jump-start":
	# it wakes the orchestration so that normal replay re-emits
	# scheduleTask actions for the removed activities, causing the
	# previously-failed work to rerun. No further rewrite is needed,
	# so we fall through to the normal replay path below.
	has_rewind_in_new = any(
	e.HasField("executionRewound") for e in new_events
	)
	if has_rewind_in_new and any(e.HasField("executionCompleted") for e in old_events):
	# The orchestration completed (with failure) and needs
	# rewinding — short-circuit to build clean history.
	return self._build_rewind_result(
	instance_id, orchestration_name, old_events, new_events)
	ctx = _RuntimeOrchestrationContext(
	instance_id,
	self._registry,
	maximum_timer_interval=self._maximum_timer_interval,
	data_converter=self._data_converter,
	)
	try:
	# Rebuild local state by replaying old history into the orchestrator function
	self._logger.debug(
	f"{instance_id}: Rebuilding local state with {len(old_events)} history event..."
	)
	ctx._is_replaying = True # pyright: ignore[reportPrivateUsage]
	for old_event in old_events:
	self.process_event(ctx, old_event)

TaskHubGrpcWorker._execute_orchestrator() constructs list(req.pastEvents) + list(req.newEvents) only to find the execution-started event. _OrchestrationExecutor.execute() then builds a list of all execution-started events from old_events, scans new and old events for rewind state, and later replays all old events.

A potential or proposed solution

Use lazy iteration (itertools.chain) for trace extraction and avoid materializing lists when only the first matching event is required. Consolidate metadata and rewind detection into minimal passes where semantics permit. Preserve rewind behavior, replay order, and non-determinism detection.