Skip to content

feat: Update LangChain runners to implement Runner protocol returning RunnerResult#150

Merged
jsonbailey merged 7 commits intomainfrom
jb/aic-2388/langchain-runner-protocol
May 1, 2026
Merged

feat: Update LangChain runners to implement Runner protocol returning RunnerResult#150
jsonbailey merged 7 commits intomainfrom
jb/aic-2388/langchain-runner-protocol

Conversation

@jsonbailey
Copy link
Copy Markdown
Contributor

@jsonbailey jsonbailey commented Apr 28, 2026

Summary

  • `LangChainModelRunner` and `LangChainAgentRunner` now formally inherit from the `Runner` protocol class.
  • `LangChainModelRunner.run()` implements the unified `Runner` protocol; returns `RunnerResult` with `content`, `metrics`, `raw`, and `parsed` fields. Structured output is supported via the `output_type` parameter.
  • `LangChainAgentRunner.run()` updated to return `RunnerResult`; populates `tool_calls` in `LDAIMetrics` from observed tool call names in the message response.
  • Legacy `invoke_model()` and `invoke_structured_model()` removed — the deprecated type definitions (`ModelRunner`, `AgentRunner`, `ModelResponse`, etc.) are kept in `server-ai` for provider CI compatibility until a cleanup PR removes them.

Stacking

Stacked on top of `jb/aic-2388/openai-runner-protocol` (PR #149).

Test plan

  • `make test-langchain` — all LangChain provider tests pass
  • `make test` — all tests across server-ai, langchain, and openai packages pass

🤖 Generated with Claude Code


Note

Medium Risk
This is an API-shape change for LangChain runners (moving to Runner.run() + RunnerResult and removing old response types), which could break downstream integrations and metric parsing if any callers still expect the legacy methods/fields.

Overview
Updates the LangChain integration to the unified Runner protocol: LangChainModelRunner and LangChainAgentRunner now expose a single run(input, output_type=...) that returns RunnerResult (content, raw, optional parsed) instead of the legacy invoke_model/invoke_structured_model responses.

Adds structured-output support to LangChainModelRunner.run() via output_type and normalizes inputs (string vs List[LDMessage]), and enhances agent runs to capture tool call names into LDAIMetrics.tool_calls.

Docs and tests are updated to use LDAIClient.create_model/ManagedModel.run() and the new result shapes; LangGraph agent-graph running no longer returns evaluation results on AgentGraphResult (they’re only flushed/tracked internally).

Reviewed by Cursor Bugbot for commit 1e61d42. Bugbot is set up for automated code reviews on this repo. Configure here.

@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from f1845b4 to 94f09ee Compare April 29, 2026 13:14
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 37bad31 to b708885 Compare April 29, 2026 13:14
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 94f09ee to 8463109 Compare April 29, 2026 13:19
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from b708885 to c553fbd Compare April 29, 2026 13:19
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 8463109 to eacddee Compare April 29, 2026 13:22
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from c553fbd to 5df809b Compare April 29, 2026 13:22
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from eacddee to 842e4e6 Compare April 29, 2026 13:46
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 5df809b to c6e35a4 Compare April 29, 2026 13:48
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 842e4e6 to 4c95357 Compare April 29, 2026 13:56
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from c6e35a4 to 08dfcb7 Compare April 29, 2026 13:56
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 4c95357 to 3f6882c Compare April 29, 2026 14:38
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 08dfcb7 to ca37e74 Compare April 29, 2026 14:38
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 3f6882c to efeea93 Compare April 29, 2026 16:31
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from ca37e74 to 9c0003c Compare April 29, 2026 16:31
@jsonbailey jsonbailey marked this pull request as ready for review April 29, 2026 16:36
@jsonbailey jsonbailey requested a review from a team as a code owner April 29, 2026 16:36
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from efeea93 to 330acf1 Compare April 29, 2026 16:49
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 9c0003c to 1aa1069 Compare April 29, 2026 16:49
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 330acf1 to cd983aa Compare April 29, 2026 17:13
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 1aa1069 to f811cf8 Compare April 29, 2026 17:13
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from cd983aa to 184be64 Compare April 29, 2026 17:14
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from f811cf8 to e56ea8f Compare April 29, 2026 17:15
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 184be64 to 4138d3c Compare April 29, 2026 17:18
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from e56ea8f to 4e0b78d Compare April 29, 2026 17:18
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 4138d3c to 7df7854 Compare April 29, 2026 17:37
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 4e0b78d to c1c2c8c Compare April 29, 2026 17:37
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from a770b1f to 069c0ee Compare April 29, 2026 21:13
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from dc86b07 to 2ba8406 Compare April 29, 2026 21:56
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 069c0ee to 1c0255f Compare April 29, 2026 21:56
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 2ba8406 to 9254ccc Compare April 30, 2026 13:59
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 1c0255f to 0355872 Compare April 30, 2026 14:01
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 9254ccc to d113d46 Compare April 30, 2026 14:20
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 0355872 to 796bda5 Compare April 30, 2026 14:21
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from d113d46 to 2878bda Compare April 30, 2026 14:43
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 796bda5 to a2db8cb Compare April 30, 2026 14:44
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from a2db8cb to cbc377d Compare May 1, 2026 14:20
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit cbc377d. Configure here.

@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 503c7b7 to 06e324f Compare May 1, 2026 16:33
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from cbc377d to f53f119 Compare May 1, 2026 16:37
@jsonbailey jsonbailey force-pushed the jb/aic-2388/openai-runner-protocol branch from 06e324f to b6d362a Compare May 1, 2026 16:41
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 34598f5 to 36cd239 Compare May 1, 2026 17:28
Base automatically changed from jb/aic-2388/openai-runner-protocol to main May 1, 2026 21:20
jsonbailey and others added 7 commits May 1, 2026 16:28
…unner]

The factory's downstream consumers (ManagedModel, ManagedAgent) now
take Runner; aligning the factory's return types lets us drop the
type: ignore comments at the ManagedModel/ManagedAgent call sites.
Provider package PRs will update their concrete implementations to
match.

Judge still takes ModelRunner, so its call site picks up the
type: ignore[arg-type] in its place — that's resolved later in the
cleanup PR when Judge migrates to Runner.
- Judge now accepts Runner instead of ModelRunner
- evaluate() calls runner.run(output_type=...) instead of invoke_structured_model
- response.parsed replaces StructuredResponse.data; None guard added
- evaluate_messages() accepts RunnerResult instead of ModelResponse
- Tests updated to use RunnerResult and mock_runner.run

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… RunnerResult

- LangChainModelRunner.run() implements the unified Runner protocol; returns RunnerResult
  with content, metrics (LDAIMetrics), raw, and parsed fields. Structured output is
  supported via the output_type parameter.
- LangChainAgentRunner.run() updated to return RunnerResult; populates tool_calls in
  LDAIMetrics from observed tool_calls in message responses.
- Legacy invoke_model() and invoke_structured_model() retained as deprecated adapters
  that delegate to run() and wrap results into ModelResponse / StructuredResponse for
  backward compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rit Runner

- LangChainModelRunner: replaces invoke_model/invoke_structured_model with
  run(input, output_type=None); returns RunnerResult
- LangChainAgentRunner: replaces AgentResult with RunnerResult; run()
  signature gains optional output_type parameter
- Tests updated to call run() and assert result.content / result.parsed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tion in LangChainAgentRunner, add _coerce_input comment
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jsonbailey jsonbailey force-pushed the jb/aic-2388/langchain-runner-protocol branch from 7aae3e2 to 1e61d42 Compare May 1, 2026 21:28
@jsonbailey jsonbailey merged commit 62a8e25 into main May 1, 2026
45 checks passed
@jsonbailey jsonbailey deleted the jb/aic-2388/langchain-runner-protocol branch May 1, 2026 21:34
@github-actions github-actions Bot mentioned this pull request May 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants