feat: add model test harness for validating cog SDK releases

[markphelps]

Summary

• Adds a declarative test harness (tools/test-harness/) that automates building and running cog models against new SDK versions
• Covers all 10 models in replicate/cog-examples and is designed to be extensible to any model from any repo
• Models, inputs, and expected outputs are defined in manifest.yaml — adding a new model requires zero code changes

How it works

1. Clones the target repo (shallow clone, cached per run)
2. Patches cog.yaml to inject sdk_version (e.g. 0.17.0rc2)
3. Runs cog build and captures build logs
4. Runs cog predict / cog train with the specified inputs
5. Validates output using pluggable validators (exact, contains, file_exists, json_match, etc.)
6. Produces a console or JSON report

Usage

cd tools/test-harness
python3 -m venv .venv && source .venv/bin/activate && pip install pyyaml

# List models
python -m harness list

# Run all non-GPU models
python -m harness run --no-gpu

# Run a specific model
python -m harness run --model hello-world

# JSON report
python -m harness run --no-gpu --output json --output-file results/report.json

Initial test results (0.17.0-rc2 SDK, non-GPU)

+ hello-world         (12.6s build, 9.8s predict)
+ canary              (12.0s build, 9.3s predict)
+ blur                (9.3s build, 8.6s predict)
+ hello-image         (10.1s build, 8.6s predict)
x hello-concurrency   BUILD FAIL — emit_metric not exported from RC SDK
x hello-context       PREDICT FAIL — current_scope().context missing in coglet
x hello-train         BUILD FAIL — transient DNS error (passes on retry)
- hello-replicate     SKIPPED — no REPLICATE_API_TOKEN

5/8 passed, 1 skipped, 2 FAILED (1 transient)

Two real compatibility issues found:

1. emit_metric is not importable from cog in the RC SDK (breaks hello-concurrency) fix(sdk): restore emit_metric as deprecated compat shim #2850
2. Scope.context attribute doesn't exist on coglet's _sdk.Scope (breaks hello-context) feat(coglet): restore Scope.context for per-prediction context #2853

Files

File	Purpose
manifest.yaml	Declarative test definitions for all 10 cog-examples models
harness/cli.py	CLI entry point (run, build, list commands)
harness/runner.py	Core loop: clone, patch, build, predict, validate
harness/patcher.py	Injects sdk_version and overrides into cog.yaml
harness/validators.py	7 validation strategies
harness/report.py	Console + JSON report generation
fixtures/	Test images for blur and resnet

[michaeldwan]

nice!