Perf: avoid repeated orchestration-history scans for tracing - #788
Open
berndverst wants to merge 2 commits into
Open
Perf: avoid repeated orchestration-history scans for tracing#788berndverst wants to merge 2 commits into
berndverst wants to merge 2 commits into
Conversation
OnRunOrchestratorAsync previously scanned orchestration history to correlate tracing events unconditionally, and for every qualifying new event it rescanned the full past-events list to find the originating TaskScheduled/SubOrchestrationInstanceCreated event - O(new events x past events) work, even when no tracing listener was registered. - Add TraceHelper.HasListeners, a cheap check backed by ActivitySource.HasListeners(), and gate all tracing-correlation work in OnRunOrchestratorAsync behind it so nothing is scanned when no listener is registered. - Replace the LINQ Concat-based scan for the ExecutionStarted event with a single-pass local function. - Add TraceHistoryEventLookup, which builds one dictionary index per work item (built lazily, once) for O(1) repeated lookups instead of rescanning PastEvents per new event, while preserving the exact original first-wins (SubOrchestrationInstanceCreated) and last-wins (TaskScheduled) semantics for duplicate event IDs. No public API changes; trace output, error handling, and replay determinism are unaffected. Adds regression tests for the no-listener fast path, first/last-wins correlation semantics end-to-end, and direct unit tests for TraceHistoryEventLookup. Fixes #769 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9c538f3e-308d-48a3-af7f-b3baf90e97b2
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Improves the gRPC worker’s tracing correlation performance by avoiding unnecessary orchestration-history scans when tracing is disabled and by indexing past history events for O(1) correlation lookups when tracing is enabled. This aligns with the SDK’s goal of keeping worker-side orchestration processing efficient, especially for long-running instances with large histories.
Changes:
- Added a cheap listener check (
TraceHelper.HasListeners) and gated trace-correlation work inOnRunOrchestratorAsyncbehind it. - Replaced repeated past-history scans with a per-work-item indexed lookup (
TraceHistoryEventLookup) that preserves prior first/last-wins semantics for duplicate event IDs. - Added unit and worker-level tests covering listener/no-listener behavior and duplicate-ID correlation semantics.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs | Adds focused unit tests for the new indexed lookup, including duplicate-ID first/last-wins semantics and filtering by event type. |
| test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs | Adds end-to-end worker tests for the no-listener fast path and for correlation semantics under a real ActivityListener. |
| src/Worker/Grpc/GrpcDurableTaskWorker.Processor.cs | Gates tracing correlation behind HasListeners, replaces LINQ concat scan with a single-pass helper, and uses the indexed lookup for repeated correlation. |
| src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs | Introduces a lazily-built, cached dictionary index for past events to avoid O(N×M) repeated scans. |
| src/Shared/Grpc/Tracing/TraceHelper.cs | Exposes HasListeners backed by ActivitySource.HasListeners() to allow callers to skip trace-correlation work when tracing is disabled. |
…history-scan-perf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OnRunOrchestratorAsyncinsrc/Worker/Grpc/GrpcDurableTaskWorker.Processor.csscanned orchestration history to correlate tracing events unconditionally, and rescanned the full past-events list for every qualifying new event to find the originatingTaskScheduled/SubOrchestrationInstanceCreatedevent — O(new events × past events), even when no tracing listener was registered.This is a precise, internal-only performance fix:
TraceHelper.HasListeners, a cheap check backed byActivitySource.HasListeners(), and gated all tracing-correlation work inOnRunOrchestratorAsyncbehind it, so nothing is scanned when no listener is registered.Concat-based scan for theExecutionStartedevent with a single-pass local function.TraceHistoryEventLookup(src/Shared/Grpc/Tracing/TraceHistoryEventLookup.cs), which builds one dictionary index per work item (lazily, at most once per event type) for O(1) repeated lookups instead of rescanningPastEventsper new event, while preserving the exact original first-wins (SubOrchestrationInstanceCreated) and last-wins (TaskScheduled) semantics for duplicate event IDs.No public API changes. Trace output, error handling, and orchestration replay determinism are unaffected — this only changes how existing history is looked up, not what is looked up or emitted.
Tests
test/Worker/Grpc.Tests/TraceHistoryEventLookupTests.cs(new): direct unit tests forTraceHistoryEventLookupcovering duplicate-event-ID first/last-wins semantics, no-match, event-type filtering, and empty input.test/Worker/Grpc.Tests/GrpcDurableTaskWorkerTests.cs(added):ActivityListenerand asserts the resulting activities reflect the original first/last-wins correlation semantics.Worker.Grpc.Testssuite passes (145/145).Worker.Grpcbuilds cleanly across all target frameworks (netstandard2.0;net6.0;net8.0;net10.0).Fixes #769