Skip to content

feat: LLM-friendly output formats (TAP + JSON)#382

Merged
ngarbezza merged 13 commits into
mainfrom
380-llm-friendly-output
Jun 16, 2026
Merged

feat: LLM-friendly output formats (TAP + JSON)#382
ngarbezza merged 13 commits into
mainfrom
380-llm-friendly-output

Conversation

@ngarbezza

Copy link
Copy Markdown
Owner

Implements #380 — the first quick win of the AI roadmap (see doc/ai-ideas.md). Adds machine-readable output so an LLM agent (or any tool) can consume Testy's results with few tokens. Driver: simplicity as cost.

What's included

  • Polymorphic Formatter family: a base Formatter (event protocol + protected helpers, no-op defaults) with three subclasses:
    • ConsoleFormatter — the existing human-readable colored output (default, unchanged).
    • TapFormatter — TAP version 13, streaming one line per test.
    • JsonFormatter — a single structured JSON object printed at the end.
  • FormatterFactory selects the formatter; ConsoleUI wires it from configuration.output().
  • Configuration surface: output key in .testyrc.json (default console) and -o/--output <console|tap|json> CLI flag, with validation. --help updated.
  • Docs: README.md + README_es.md (config option, CLI flag, "machine-readable output" section).

Machine output is language-neutral (statuses/keys are not translated); only failure messages are localized to the configured language. AI lives entirely outside the core — zero new runtime dependencies.

Design / plan

  • Design doc: doc/plans/2026-06-06-llm-friendly-output-design.md
  • Implementation plan: doc/plans/2026-06-06-llm-friendly-output.md

Examples

npx testy -o tap
npx testy -o json

TAP:

TAP version 13
ok 1 - something is true
not ok 2 - sum works
  ---
  message: Expected 5 to be equal to 4
  at: tests/example_test.js:12
  severity: failure
  ---
1..2
# tests 2
# pass 1
# fail 1
...

JSON (single line): { "tool": "@pmoo/testy", "version": "...", "summary": { total, passed, failed, errored, pending, skipped, durationMs }, "suites": [ { name, file, tests: [ { name, status, failure? } ] } ] }

Testing

  • TDD throughout; new tests for TAP, JSON, the factory, the config accessor, and the CLI flag (incl. the ordering fix + invalid-value rejection).
  • Full suite: 405/405 passing; lint clean.
  • End-to-end verified via the real CLI for -o tap, -o json, and default.

Known limitations / follow-ups (non-blocking)

  • TAP failure message: collapses internal newlines to spaces (error stack traces become one line).
  • An invalid output value coming from .testyrc.json (not the CLI) silently falls back to console, matching how language is handled today.
  • displayError (top-level runtime errors during a run) is a no-op in TAP/JSON mode — machine consumers rely on the non-zero exit code; emitting a Bail out! / {"error":...} could be a follow-up.
  • Nice-to-have tests: multi-suite JSON accumulation, explicit TAP stack-collapsing/# time assertions.

Closes #380.

🤖 Generated with Claude Code

ngarbezza and others added 13 commits June 11, 2026 21:40
Approved design for TAP + JSON output formats: polymorphic Formatter
family, output config key + -o/--output CLI flag, output contracts,
testing and docs plan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bite-sized TDD plan: base Formatter refactor, TapFormatter,
JsonFormatter, FormatterFactory, output config + CLI flag, docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure refactor under green tests: base Formatter defines the event
protocol with no-op defaults and protected helpers; ConsoleFormatter
holds the existing human output unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Also drive the run timer in the formatter test helper so duration is realistic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CLI ordering validation only whitelisted -d/-e as value-taking
flags, so a value following -o/--output was rejected as a stray path
param. Include the output identifiers in the whitelist and add a
regression test through getPathsAndConfigurationParams.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- CLAUDE.md documents commands, architecture, conventions, and extension
  points for AI-assisted development in this repo
- doc/plans/ added to .gitignore; AI-generated implementation plans are
  ephemeral and should not be part of the permanent history
- Remove the two plan files from the index (files remain on disk)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extracts the text-collapsing logic from TapFormatter into a reusable
normalizeToSingleLine() function in lib/utils/formatting.js, per ADR-0002.
Uses split/join instead of regex to avoid the ReDoS hotspot (S5852)
flagged by SonarCloud. Adds 5 tests covering the new utility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ngarbezza ngarbezza merged commit 950259d into main Jun 16, 2026
5 checks passed
@ngarbezza ngarbezza deleted the 380-llm-friendly-output branch June 16, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI roadmap (fase 0): salida/reporter LLM-friendly

1 participant