Skip to content

v0.18.0

Choose a tag to compare

@mortenoh mortenoh released this 22 Apr 12:19
· 133 commits to main since this release
v0.18.0
7d93209

Summary

New ML validation endpoint (CLIM-581): POST /api/v1/ml/\$validate accepts the same payloads as \$train / \$predict (distinguished by a type discriminator) and returns structured ValidationDiagnostic entries without running a job. Moves model-validation knowledge into chapkit so the frontend sends the payload it already has and reacts to structured diagnostics.

What shipped

  • Endpoint: POST /api/v1/ml/\$validate with Pydantic discriminated union on type (matches existing ArtifactData convention). Returns 200 OK with {valid: bool, diagnostics: [...]}.
  • Framework-level automatic checks (always-on, no user code needed): config/artifact existence, prediction_periods bounds, failed training artifacts, empty data, workspace ZIP integrity, model.pickle corruption.
  • Runner hooks: on_validate_train / on_validate_predict -- optional callbacks on all three runner types (Functional, Shell, class-based). Hooks are only invoked when no framework error-severity diagnostic was raised, so hooks can safely assume inputs are structurally valid.
  • ValidationDiagnostic classmethods: .error(), .warning(), .info() for cleaner construction (replaces verbose severity= kwarg).
  • CLI: chapkit init --with-validation scaffolds both hooks with richly-documented stubs wired into the runner. Off by default. Rejected for --template task (no ML runner). Postman collections always include \$validate regardless of the flag.
  • Docs: endpoint reference, diagnostic schema, classmethods, hook examples (functional/shell/class) in `ml-workflows.md`; `--with-validation` option and "With Validation Hooks" subsection in `cli-scaffolding.md`.
  • Example: `examples/ml_class/main.py` demonstrates `on_validate_train` with `insufficient_training_samples`.

Design decisions

  • One endpoint with discriminated union (not two separate endpoints) -- single URL/operation for the FE, response shape identical for both flows.
  • Synchronous -- no scheduler, no artifacts written.
  • Runner hook short-circuit -- framework errors stop hook invocation so hooks cannot crash on invalid inputs.
  • Workspace integrity -- `$validate` mirrors the deterministic checks in `_predict_task` (ZIP, model.pickle load) so it never reports `valid=True` for payloads `$predict` would reject.
  • Shell-runner validate hooks run in Python -- they inspect the payload directly rather than invoking shell scripts, keeping `$validate` fast.

Full PR: #33