lex-validator

Fleet pipeline validation for LegionIO. Receives implemented work items and runs them through a four-stage gauntlet — tests, lint, security scan, and adversarial multi-model LLM review — before rendering a verdict.

Overview

lex-validator is the fourth and final stage of the LegionIO Fleet Pipeline:

lex-assessor → lex-planner → lex-developer → lex-validator → ship
                                                    ↓
                                          (rejected) → lex-developer (feedback loop)

When a work item arrives, the validator runs each check in sequence. All checks must pass. Any failure produces a rejected verdict with structured feedback routed back to the developer for the next iteration. A passing item advances to the ship stage.

Validation Pipeline

Step	Check	Early Exit
1	Tests — bundle exec rspec via lex-exec	No
2	Lint — bundle exec rubocop via lex-exec	No
3	Security scan — per-file pattern analysis	No
4	PR comment check — filter out unresolved external review comments	Yes — on unresolved human feedback
5	Stale diff guard — verify PR head SHA matches developer push	Yes — on SHA mismatch
6	Adversarial LLM review — k independent reviewers, unanimity required	No
7	Quality gate — weighted scoring, threshold ≥ 0.8	No

Quality Gate

Scores are composed from the check results and weighted:

Dimension	Weight	Signal
Completeness	35%	Test pass rate
Correctness	35%	Lint clean + security clean
Quality	20%	LLM review verdict
Security	10%	Security scan clean

Weights and threshold are configurable via Legion::Settings:

fleet:
  validation:
    quality_gate_threshold: 0.8
    quality_weights:
      completeness: 0.35
      correctness: 0.35
      quality: 0.20
      security: 0.10

Adversarial Review

lex-validator runs k independent LLM reviewers in parallel (configurable per work item, default from implementation.validators). Unanimity is required — a single dissenting vote rejects the work item. Errors and timeouts also reject (fail closed).

Each reviewer receives a different intent.capability level (basic, moderate, reasoning) to encourage model diversity across the review pool (MAKER approach). Upstream models from the pipeline trace are excluded to prevent same-model bias.

fleet:
  llm:
    validator_timeout_seconds: 120

Installation

gem 'lex-validator'

Usage

require 'legion/extensions/validator'

result = Legion::Extensions::Validator::Runners::Validator.validate(
  work_item: {
    work_item_id: 'uuid-001',
    title: 'Fix sandbox timeout on macOS',
    repo: { owner: 'LegionIO', name: 'lex-exec', language: 'ruby' },
    config: {
      estimated_difficulty: 0.5,
      implementation: { validators: 3 },
      validation: {
        enabled: true,
        run_tests: true,
        run_lint: true,
        security_scan: true,
        adversarial_review: true
      }
    },
    pipeline: {
      stage: 'implemented',
      trace: [],
      attempt: 0,
      feedback_history: [],
      changes: ['lib/sandbox.rb'],
      pr_number: 42,
      branch_name: 'fleet/fix-lex-exec-42'
    }
  }
)

result[:success]                                    # => true
result[:work_item][:pipeline][:stage]               # => 'validated'
result[:work_item][:pipeline][:review_result]       # => { verdict: 'approved', score: { aggregate: 0.95, ... }, ... }
result[:work_item][:pipeline][:trace].last[:stage]  # => 'validator'

Verdict shape

{
  verdict:        'approved',   # or 'rejected'
  score: {
    completeness: 1.0,
    correctness:  1.0,
    quality:      1.0,
    security:     1.0,
    aggregate:    1.0
  },
  issues:          [],          # strings describing failures
  merged_feedback: []           # deduplicated feedback for developer
}

On rejection, pipeline.feedback_history is appended with the round's issues, enabling the developer stage to incorporate targeted feedback on the next attempt.

Disabling checks

Any validation check can be disabled individually:

validation: {
  enabled: true,
  run_tests: false,        # skip rspec
  run_lint: false,         # skip rubocop
  security_scan: false,    # skip security scan
  adversarial_review: false  # skip LLM review (quality score defaults to 1.0)
}

Set enabled: false to skip validation entirely and return an immediate approved verdict.

Transport

• Exchange: lex.validator (topic, durable)
• Queue: lex.validator.runners.validator (durable, routing key lex.validator.runners.validator.#)

Development

bundle install
bundle exec rspec        # 48 examples
bundle exec rubocop      # 0 offenses

License

MIT