Skip to content

feat: add ProfilesClient.validateProfile for LLM pre-flight checks - #337

Merged
neubig merged 1 commit into
mainfrom
feat/profiles-validate
Aug 11, 2026
Merged

feat: add ProfilesClient.validateProfile for LLM pre-flight checks#337
neubig merged 1 commit into
mainfrom
feat/profiles-validate

Conversation

@neubig

@neubig neubig commented Aug 11, 2026

Copy link
Copy Markdown
Member

Why

The agent-server is adding a pre-flight LLM validation endpoint
(POST /api/profiles/{name}/validate, software-agent-sdk PR #4422) so
LLM misconfigurations can be caught before a profile is saved. Agent
Canvas (OpenHands/OpenHands#16417) needs a typed client method to call it
instead of a generic request.

Summary

  • ProfilesClient.validateProfile(name, request)POST /api/profiles/{name}/validate.
  • New ValidateProfileResponse / ValidateProfileError models exported from the package root.
  • Endpoint-audit allowlist entry with tracking to software-agent-sdk PR #4422 (the endpoint is client-ahead of the pinned agent-server release).

Issue Number

How to Test

npm ci
npm run build
npm run lint
npm test -- --testPathPatterns=api-clients

Unit coverage: success verdict, valid:false error verdict, name
percent-encoding, and 404 surfacing as HttpError.

Video/Screenshots

N/A — library API addition; no UI.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Draft: this method depends on the agent-server endpoint from
    software-agent-sdk PR #4422, which is not yet in a released agent-server
    image. The endpoint audit is report-only for this client-ahead API.
  • A new @openhands/typescript-client release is required before
    dependent consumers can switch off the git-SHA pin.

This PR was created by an AI agent (OpenHands) on behalf of the user.

Adds a typed ProfilesClient.validateProfile() that calls the agent-server
POST /api/profiles/{name}/validate endpoint, firing a minimal 1-token LLM
completion to catch misconfigurations (invalid model names, missing
provider prefixes, bad base URLs, invalid API keys) before a profile is
saved. The endpoint is client-ahead of the pinned agent-server release
(software-agent-sdk PR #4422), so it is allowlisted in the endpoint audit
with tracking.

Adds ValidateProfileError / ValidateProfileResponse models, exports them
from the package root, and covers the method with jest tests (success,
valid=false verdict, name encoding, 404 HttpError).

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

Copy link
Copy Markdown
Contributor

Endpoint audit

⚠️ 7 actionable Agent Server contract divergence(s) · report-only

Contract: pinned release artifact

Category Count
Actionable client-only calls 0
Actionable server-only operations 7
Documented non-divergences 13
Agent Server contract operations 117
Audited handwritten client endpoints 119

Actionable client-only calls (0)

none

Actionable server-only operations (7)

  • GET /api/conversations/{}/events
  • GET /api/file/archive
  • GET /api/git/commits
  • GET /api/git/commits/{}/changes
  • GET /api/init
  • POST /api/conversations/{}/load_plugin
  • POST /api/init
Documented non-divergences (13)

Client calls intentionally absent from the filtered contract (11)

  • GET /
  • GET /alive
  • GET /health
  • GET /ready
  • GET /server_info

Reason: Operational Agent Server endpoints intentionally excluded from the filtered public release artifact.
Owner: OpenHands runtime maintainers

  • DELETE /api/meta-profiles/{}
  • GET /api/meta-profiles
  • GET /api/meta-profiles/{}
  • POST /api/meta-profiles/{}
  • POST /api/meta-profiles/{}/activate

Reason: Client-ahead API stacked on the pending Agent Server meta-profiles implementation.
Owner: OpenHands SDK maintainers
Tracking: OpenHands/software-agent-sdk#3744

  • POST /api/profiles/{}/validate

Reason: Client-ahead API stacked on the pending Agent Server pre-flight LLM validation endpoint.
Owner: OpenHands TypeScript client maintainers
Tracking: OpenHands/software-agent-sdk#4422

Server operations covered by an exposed browser URL (2)

  • GET /api/conversations/{}/workspace
  • GET /api/conversations/{}/workspace/{}

Reason: RemoteWorkspace.startWorkspaceSession exposes these authenticated URLs for browser iframe and file requests; they are not HttpClient method calls.
Owner: OpenHands TypeScript client maintainers

@github-actions github-actions Bot added the type: feat A new feature label Aug 11, 2026
neubig added a commit to OpenHands/OpenHands that referenced this pull request Aug 11, 2026
Switch ProfilesService.validateProfile() from a generic AgentServerClient
request to the typed @openhands/typescript-client ProfilesClient method
added for the agent-server pre-flight validation endpoint. Pin the client
dependency to the git SHA carrying the new API; 404 from older
agent-server versions is still treated as "no verdict".

NOTE: dependency pin is temporary — agent-canvas policy requires a
released @openhands/typescript-client version. Rebase onto the release
once the client PR (OpenHands/typescript-client#337) publishes.

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig marked this pull request as ready for review August 11, 2026 15:46

@VascoSch92 VascoSch92 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@neubig
neubig merged commit 0786979 into main Aug 11, 2026
12 checks passed
@openhands-release-bot openhands-release-bot Bot added the released: v1.38.0 Shipped in v1.38.0 label Aug 12, 2026
@openhands-release-bot

Copy link
Copy Markdown
Contributor

🚀 Released in v1.38.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

released: v1.38.0 Shipped in v1.38.0 type: feat A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants