Skip to content

feat: Update OpenAI graph runner to return AgentGraphRunnerResult with GraphMetrics#155

Merged
jsonbailey merged 5 commits intomainfrom
jb/aic-2174/openai-graph-runner
May 4, 2026
Merged

feat: Update OpenAI graph runner to return AgentGraphRunnerResult with GraphMetrics#155
jsonbailey merged 5 commits intomainfrom
jb/aic-2174/openai-graph-runner

Conversation

@jsonbailey
Copy link
Copy Markdown
Contributor

@jsonbailey jsonbailey commented Apr 28, 2026

Summary

  • Removes all direct LaunchDarkly tracker calls from OpenAIAgentGraphRunner
  • Introduces _NodeMetricsAccumulator — a lightweight per-node metrics collector replacing LDAIConfigTracker inside the runner
  • Runner now returns AgentGraphRunnerResult with populated GraphMetrics (path, duration_ms, usage, node_metrics)
  • Graph-level and per-node tracking events are emitted by ManagedAgentGraph._flush_graph_tracking() from the result metrics
  • ManagedAgentGraph._flush_graph_tracking() extended to drive per-node tracking from result.metrics.node_metrics using graph node tracker factories
  • Integration tests updated to exercise the full ManagedAgentGraph.run() pipeline (tracking events now come from the managed layer)
  • Handoff-level track_handoff_success() calls removed (per spec: path field is sufficient; handoffs are not in GraphMetrics)

Depends on

Test plan

  • All existing tests pass (uv run pytest packages/ai-providers/server-ai-openai/tests/)
  • test_openai_agent_graph_runner.py: runner returns new shape, no tracker created
  • test_tracking_openai_agents.py: graph-level and per-node events emitted through managed layer

🤖 Generated with Claude Code


Note

Medium Risk
Changes the runner result shape and shifts LaunchDarkly event emission from the OpenAI runner to ManagedAgentGraph, which could affect downstream integrations and metric correctness if node/path/usage attribution is wrong.

Overview
OpenAIAgentGraphRunner now returns AgentGraphRunnerResult with GraphMetrics (path, duration_ms, usage, and per-node node_metrics) and removes all direct LaunchDarkly tracker calls, including handoff tracking.

Per-node durations/usage/tool calls are accumulated inside the runner (_node_metrics) during handoffs and final segment flush, and the integration path is updated so ManagedAgentGraph emits graph + node tracking events by consuming result.metrics.node_metrics and creating node trackers on demand.

Tests are updated to assert the new result fields, ensure trackers are no longer created/called by the runner, and to run tracking assertions through ManagedAgentGraph.run() (including failure cases and multi-node runs).

Reviewed by Cursor Bugbot for commit fa936cf. Bugbot is set up for automated code reviews on this repo. Configure here.

@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from fcbcb18 to 9286d53 Compare April 29, 2026 13:15
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 9733a28 to 44501e3 Compare April 29, 2026 13:15
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from 9286d53 to 72fc13e Compare April 29, 2026 13:19
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 44501e3 to 142e041 Compare April 29, 2026 13:19
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from 72fc13e to bde4f09 Compare April 29, 2026 13:22
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 142e041 to fb3c0f6 Compare April 29, 2026 13:22
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from bde4f09 to c376011 Compare April 29, 2026 13:52
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from fb3c0f6 to b3547b0 Compare April 29, 2026 13:52
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from c376011 to 7f67e4f Compare April 29, 2026 13:57
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from b3547b0 to 1d4ddb2 Compare April 29, 2026 13:57
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from 7f67e4f to a89c6a2 Compare April 29, 2026 14:38
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 1d4ddb2 to 6201d09 Compare April 29, 2026 14:38
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from a89c6a2 to c69a9ff Compare April 29, 2026 16:33
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 6201d09 to ef4216c Compare April 29, 2026 16:34
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from c69a9ff to 14cfa92 Compare April 30, 2026 14:03
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from ef4216c to 8ecce16 Compare April 30, 2026 14:05
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from 14cfa92 to 1ed1a44 Compare April 30, 2026 14:23
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch 2 times, most recently from 8ecce16 to 09af502 Compare April 30, 2026 14:23
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch from 1ed1a44 to f016b0d Compare April 30, 2026 14:46
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 09af502 to 43bc879 Compare April 30, 2026 14:47
@jsonbailey jsonbailey force-pushed the jb/aic-2174/graph-tracking-refactor branch 3 times, most recently from 5527667 to 8035890 Compare May 4, 2026 15:14
Base automatically changed from jb/aic-2174/graph-tracking-refactor to main May 4, 2026 16:56
…h GraphMetrics

Remove all direct LaunchDarkly tracker calls from OpenAIAgentGraphRunner. The
runner now collects per-node metrics via _NodeMetricsAccumulator (a lightweight
accumulator replacing the per-node LDAIConfigTracker) and returns
AgentGraphRunnerResult with populated GraphMetrics (path, duration_ms, usage,
node_metrics). Graph-level and per-node tracking events are emitted by
ManagedAgentGraph from the result via track_graph_metrics_of_async and
_track_node_metrics.

Integration tests in test_tracking_openai_agents.py are updated to run through
the full ManagedAgentGraph pipeline (ManagedAgentGraph.run()) so tracking events
are emitted by the managed layer as intended.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jsonbailey jsonbailey force-pushed the jb/aic-2174/openai-graph-runner branch from 43bc879 to f6df746 Compare May 4, 2026 17:10
@jsonbailey jsonbailey marked this pull request as ready for review May 4, 2026 18:57
@jsonbailey jsonbailey requested a review from a team as a code owner May 4, 2026 18:57
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 49cb245. Configure here.

@jsonbailey jsonbailey merged commit 388b7af into main May 4, 2026
45 checks passed
@jsonbailey jsonbailey deleted the jb/aic-2174/openai-graph-runner branch May 4, 2026 21:05
@github-actions github-actions Bot mentioned this pull request May 4, 2026
jsonbailey added a commit that referenced this pull request May 6, 2026
🤖 I have created a release *beep* *boop*
---


<details><summary>launchdarkly-server-sdk-ai: 0.19.0</summary>

##
[0.19.0](launchdarkly-server-sdk-ai-0.18.0...launchdarkly-server-sdk-ai-0.19.0)
(2026-05-05)


### ⚠ BREAKING CHANGES

* StructuredResponse replaced by RunnerResult with new "parsed" property
* AgentResult replaced by RunnerResult and Managed Result
* Removed ModelRunner and AgentRunner protocols
* Removed invoke_method, invoke_structured_model from AIProvider base
class.
* ModelResponse was replaced by RunnerResult
* Add ManagedResult, RunnerResult, and Runner protocol; rename invoke()
to run()
([#148](#148))
* Swap track_metrics_of parameter order to match spec
([#144](#144))

### Features

* Add evaluations support to ManagedAgent.run()
([#153](#153))
([442f46a](442f46a))
* Add judge evaluation support to agent graphs
([#142](#142))
([3d5a6a9](3d5a6a9))
* Add ManagedGraphResult, GraphMetricSummary, and AgentGraphRunnerResult
types
([#151](#151))
([301e24c](301e24c))
* Add ManagedResult, RunnerResult, and Runner protocol; rename invoke()
to run()
([#148](#148))
([88d4ddc](88d4ddc))
* Add root-level tools map with customParameters to AI Config types
([#141](#141))
([f17c535](f17c535))
* bake sampling_rate into Judge at construction; simplify Evaluator to
List[Judge]
([#159](#159))
([86c79e6](86c79e6))
* Update LangChain runners to implement Runner protocol returning
RunnerResult
([#150](#150))
([62a8e25](62a8e25))


### Bug Fixes

* Add runtime DeprecationWarnings to deprecated methods
([#145](#145))
([2189b81](2189b81))
* AgentResult replaced by RunnerResult and Managed Result
([fbb0b4b](fbb0b4b))
* build judge input as string; strip legacy judge config messages
([#165](#165))
([e6942a6](e6942a6))
* Fall back to model.parameters.tools when root tools absent
([#146](#146))
([2c30d75](2c30d75))
* Graph tracking refactor — ManagedAgentGraph drives tracking for new
runner shape
([#154](#154))
([20a5020](20a5020))
* ModelResponse was replaced by RunnerResult
([fbb0b4b](fbb0b4b))
* parse model.parameters.tools as list
([#160](#160))
([fb53e99](fb53e99))
* reference correct PyPI package names in provider load error messages
([#164](#164))
([48761c9](48761c9))
* Removed invoke_method, invoke_structured_model from AIProvider base
class.
([fbb0b4b](fbb0b4b))
* Removed ModelRunner and AgentRunner protocols
([fbb0b4b](fbb0b4b))
* Replace done_callback with coroutine chain for judge tracking
([#147](#147))
([1e1f36b](1e1f36b))
* StructuredResponse replaced by RunnerResult with new "parsed" property
([fbb0b4b](fbb0b4b))
* Swap track_metrics_of parameter order to match spec
([#144](#144))
([53db736](53db736))
</details>

<details><summary>launchdarkly-server-sdk-ai-langchain: 0.6.0</summary>

##
[0.6.0](launchdarkly-server-sdk-ai-langchain-0.5.0...launchdarkly-server-sdk-ai-langchain-0.6.0)
(2026-05-05)


### Features

* Add judge evaluation support to agent graphs
([#142](#142))
([3d5a6a9](3d5a6a9))
* Migrate LangGraph runner to AgentGraphRunnerResult; clean up legacy
shape detection
([#156](#156))
([efa8e00](efa8e00))
* Support conversation history directly in AI Provider model runners
([#166](#166))
([4bb3e78](4bb3e78))
* Update LangChain runners to implement Runner protocol returning
RunnerResult
([#150](#150))
([62a8e25](62a8e25))


### Bug Fixes

* build judge input as string; strip legacy judge config messages
([#165](#165))
([e6942a6](e6942a6))
</details>

<details><summary>launchdarkly-server-sdk-ai-openai: 0.5.0</summary>

##
[0.5.0](launchdarkly-server-sdk-ai-openai-0.4.0...launchdarkly-server-sdk-ai-openai-0.5.0)
(2026-05-05)


### Features

* Add judge evaluation support to agent graphs
([#142](#142))
([3d5a6a9](3d5a6a9))
* Support conversation history directly in AI Provider model runners
([#166](#166))
([4bb3e78](4bb3e78))
* Update OpenAI graph runner to return AgentGraphRunnerResult with
GraphMetrics
([#155](#155))
([388b7af](388b7af))
* Update OpenAI runners to implement Runner protocol returning
RunnerResult
([#149](#149))
([382e662](382e662))


### Bug Fixes

* build judge input as string; strip legacy judge config messages
([#165](#165))
([e6942a6](e6942a6))
</details>

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Primarily a release/version bump, but it publishes **breaking API
changes** (move to unified `Runner.run()`/`RunnerResult` and removal of
`invoke_*` methods), which can break downstream integrations.
> 
> **Overview**
> Cuts a new release across the core SDK and provider packages:
`launchdarkly-server-sdk-ai` to `0.19.0`, LangChain provider to `0.6.0`,
and OpenAI provider to `0.5.0`, updating the release manifest and
package metadata accordingly.
> 
> Changelogs document the shipped breaking API surface changes (notably
removing `invoke_model()`/`invoke_structured_model()` in favor of
`run(...)` and standardizing returns on `RunnerResult`) plus
accompanying feature/fix entries; the core package version
constants/docs (`__version__`, `PROVENANCE.md`) are updated to match.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
a20d7a5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: jsonbailey <jbailey@launchdarkly.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants