fix(sdk): pack validator spec compliance (#933)

[chaholl]

Summary

Fixes #933 — the SDK silently replaced every response with DefaultBlockedMessage when a pack declared any validator, due to a JSON tag mismatch that dropped validator params.

This PR delivers the immediate fix and a set of prevention mechanisms so the class of bug cannot recur.

The bug

sdk/internal/pack.Validator had json:"config" where the promptpack spec uses json:"params", plus forbidden monitor/message fields the spec's additionalProperties:false rejects. Params silently unmarshalled to nil, every declared validator ran with empty params, the max_length handler reported "missing or zero 'max'/'max_characters' param" every turn, and the guardrail adapter replaced every response with DefaultBlockedMessage.

The pack in the issue is fully valid per the promptpack spec. The SDK was the defect.

What changed

Immediate fix (e5c66290): SDK Validator struct aligned to $defs/Validator in runtime/prompt/schema/promptpack.schema.json — type, enabled (required), fail_on_violation (optional), params (optional). convertPackValidatorsToHooks updated to skip disabled validators, honor the spec default of monitor-only when fail_on_violation is absent, and pass Params (not Config) to the guardrail factory.

Prevention mechanisms:

• Spec-parity reflection test (2538c7d3) — walks pack.Validator via reflection and diffs against the embedded schema's $defs/Validator. Any future tag rename, extra field, or required-with-omitempty fails the build.
• Schema rejection test (a02bed79) — pins that ValidateAgainstSchema rejects validators with forbidden monitor/config/message fields before struct unmarshal.
• ParamValidator interface (4edfb07b, 580cfe11) — new optional interface in runtime/evals that handlers can implement to expose required-param contracts. Implemented on MaxLengthHandler, MinLengthHandler, WorkflowStateIsHandler, GuardrailTriggeredHandler.
• Guardrail factory validates params at construction (a32e8170) — NewGuardrailHookFromRegistry normalises params (ApplyDefaults + NormalizeParams) and calls ValidateParams, so pack validators with unusable params fail at SDK load instead of silently enforcing every turn.
• ValidateEvalTypes extended (af6ad2b3) — the existing function used by Arena fail-fast and SDK eval preflight now also invokes ParamValidator. Arena gains stricter checks for free.
• SDK warn-and-skip for evals (d1e76e49) — sdk/eval_middleware.go filters bad eval defs at middleware creation with a warning, mirroring the existing validator warn-and-skip in convertPackValidatorsToHooks.
• Public sdk.ValidatePack(path, skipSchemaValidation) preflight API (8849cb5b, 2809df11, a868f1a2) — lets CI gates and operator tools check whether a pack will load cleanly before sdk.Open() is called. Strict-by-default (uses the embedded promptpack schema). Returns []PackIssue for semantic problems and error for structural/schema failures.

Tests:

• End-to-end regression (77c070ad, b88a385f) — full path sdk.Open() → pack.Load() → Send() with a mock provider, asserts content unchanged AND validator actually registered (not silently dropped by the warn-and-skip layer). Empirically verified to fail if the struct tag regresses.
• Ticket-exact reproducer (9f74362c) — literal mirror of the reproducer in Pack validators silently run with empty params due to SDK/runtime struct tag mismatch #933 (fail_on_violation: false, 25-char response).
• Arena fail-fast test (7517857c) — pins that Arena's existing fail-fast picks up ParamValidator errors.
• ValidatePack tests covering valid pack, unknown validator type, missing required params, disabled skip, unknown eval type, missing eval params, multiple issues, file-not-found, structural error, strict schema rejection of non-spec fields, and the skip-schema opt-out.

Latent bugs surfaced

The stricter validation revealed two pre-existing test-fixture bugs, both fixed in the same commits that introduced the stricter check:

• TestNewGuardrailHook_AllTypes declared max_length with only max_tokens (no max_characters) — was relying on silent runtime tolerance.
• TestEvalMiddleware_ResolvesPackAndPromptEvals used fabricated eval types (regex_override) and bare length with empty params — was bypassing validation entirely.

API addition

// strict (default — zero value)
issues, err := sdk.ValidatePack("pack.json", false)

// opt-out for tooling that wants only handler-level checks
issues, err := sdk.ValidatePack("pack.json", true)

type PackIssue struct {
    Severity string  // "error"
    Kind     string  // "validator" or "eval"
    PromptID string  // owning prompt, or "" for pack-level evals
    Index    int     // position within validators/evals slice
    Type     string  // handler type
    ID       string  // eval def ID (empty for validators)
    Reason   string  // human-readable explanation
}

Test plan

• Unit tests for ParamValidator on all four handlers with happy paths, missing keys, nil params, zero, negative, and alias key names
• Unit tests for guardrail factory happy path, missing params, unknown type, handlers without ParamValidator
• Unit tests for ValidateEvalTypes extension with stub handler
• SDK warn-and-skip filter tests (unknown type, missing params, all-valid)
• Spec-parity reflection test against embedded schema
• Schema rejection test for additionalProperties:false fields
• JSON round-trip fidelity test for Validator struct
• End-to-end happy path: short response not blocked (empirically verified to fail on struct-tag regression via log-capture assertion)
• End-to-end enforcement: long response truncated with fail_on_violation:true
• End-to-end monitor-only: long response passes through with fail_on_violation:false
• Ticket-exact reproducer matching the issue's literal JSON
• Arena fail-fast regression test for missing required params
• sdk.ValidatePack tests covering 12 scenarios including strict default and opt-out
• Full test matrix (runtime, sdk, pkg, tools/arena, tools/packc, tools/schema-gen) passes with -count=1 -race
• Lint clean (golangci-lint run --new-from-rev=main) on all changed files

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud