Skip to content

feat(models): implement structured JSON output for OpenAI#2

Merged
runtimebug merged 3 commits into
mainfrom
feat/structured-json-output
Feb 8, 2026
Merged

feat(models): implement structured JSON output for OpenAI#2
runtimebug merged 3 commits into
mainfrom
feat/structured-json-output

Conversation

@runtimebug

Copy link
Copy Markdown
Owner

Summary

Implements Phase 1 Task 3: Structured JSON output from LLM to replace fragile regex parsing.

Changes

  • New schema module (src/schemas/):

    • review-response.ts - TypeScript types for structured output (severity, category, confidence, title, explanation, suggestion)
    • review-response.json - JSON Schema for OpenAI's response_format
    • validate.ts - Validation utilities with fallback to legacy parsing
    • validate.test.ts - 33 comprehensive tests
  • Updated OpenAI adapter (src/models/openai.ts):

    • Uses json_schema response format for reliable structured output
    • Per-comment severity filtering (critical | warning | suggestion | nitpick)
    • Confidence threshold filtering (default min: 0.6)
    • Backward-compatible legacy parsing as fallback
    • Rich comment formatting with category badges and confidence scores

Why

The previous regex-based parsing (/([\w/.-]+):(\d+)\s*[—–-]\s*(.*?)(?=...)/gs) was fragile:

  • Comments were silently dropped if LLM used slightly different formatting
  • No per-comment severity or confidence scores
  • Colons in comment content could break parsing

Testing

All 94 tests pass:

  • 33 new tests for schema validation
  • 20 existing tests for OpenAI adapter (still pass with backward compatibility)
  • Full test coverage for legacy fallback mode

Backward Compatibility

The implementation gracefully falls back to legacy free-text parsing if:

  • The response doesn't look like JSON
  • JSON parsing fails
  • Validation fails

This ensures existing behavior is preserved while enabling the new structured format.

Replace regex-based response parsing with JSON Schema structured output:
- Add StructuredReviewResponse type with per-comment severity, category, confidence
- Add JSON Schema (src/schemas/review-response.json) for OpenAI response_format
- Update OpenAI adapter to use json_schema response format
- Add validation utilities for structured responses
- Maintain backward compatibility with legacy free-text parsing as fallback
- Add comprehensive tests for schema validation and conversion

This eliminates the fragile regex parsing that could silently drop comments
if the LLM formatted responses slightly differently.

Refs: Phase 1 Task 3 - Structured JSON output from LLM
…mode

OpenAI's json_schema strict mode requires ALL properties to be listed in
the 'required' array. Changed summary and suggestion from optional to
required (can be empty strings):

- Update JSON schema to include 'summary' and 'suggestion' in required
- Update TypeScript types to make fields required
- Update validation logic to check for required fields
- Update all tests to include summary and suggestion fields
- Update prompt to clarify fields are required but can be empty
@runtimebug
runtimebug merged commit a5662a2 into main Feb 8, 2026
3 checks passed
@runtimebug
runtimebug deleted the feat/structured-json-output branch February 8, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant