-
Notifications
You must be signed in to change notification settings - Fork 0
feat: add research and optimization spec contracts #231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
8151185
feat: add research and optimization spec contracts
Pigbibi 664ac0a
fix: align strategy spec schema validation
Pigbibi 4073965
fix: keep strategy spec validators schema-compatible
Pigbibi 5c20f87
fix: reject non-JSON strategy spec values
Pigbibi 0a72d2c
fix: align strategy spec validators with schemas
Pigbibi 74b0ce6
fix: enforce strategy specification invariants
Pigbibi bbbce6a
fix: complete strategy spec contract validation
Pigbibi 21d3cbe
fix: package strategy specification schemas
Pigbibi 8b79647
refactor: isolate strategy spec validation
Pigbibi e4fa557
fix: handle unbounded JSON integers
Pigbibi File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| # Quant Strategy Specification Contracts | ||
|
|
||
| This document defines the first versioned artifacts introduced by the Quant | ||
| Strategy Review and Optimization Standard. They are research evidence inputs; | ||
| they do not change existing lifecycle stages, evidence-package fields, or | ||
| runtime interfaces. | ||
|
|
||
| ## Artifacts | ||
|
|
||
| | Artifact | Schema version | Purpose | | ||
| | --- | --- | --- | | ||
| | `ResearchSpec` | `research_spec.v1` | Freeze a falsifiable hypothesis, PIT data revision, four-layer benchmarks, net cost model, OOS plan, and complete trial ledger before evaluation. | | ||
| | `OptimizationSpec` | `optimization_spec.v1` | Freeze optimization inputs, permitted parameter ranges, constrained objective, nested WFA, locked holdout, multiple-testing control, cost stress, stop rules, and human-only promotion. | | ||
|
|
||
| Schemas are stored as package data under `quant_platform_kit/schemas/`, so they | ||
| remain available in installed wheels. Python validation is deliberately | ||
| dependency free so an evidence gate can consume the same artifact without | ||
| installing a JSON Schema runtime. | ||
|
|
||
| The schemas use QPK extensions for invariants that standard JSON Schema cannot | ||
| express: sibling date ordering (`x-qpk-date-order` and | ||
| `x-qpk-exclusive-date-order`) and parameter-name uniqueness | ||
| (`x-qpk-unique-by`). QPK's validator and CLI enforce these extensions; a generic | ||
| JSON Schema-only consumer must not be used as a promotion gate. | ||
|
|
||
| ```bash | ||
| quant-strategy-spec path/to/spec.json | ||
| ``` | ||
|
|
||
| The command is silent on success, prints field-level contract violations to | ||
| stderr on failure, and returns a non-zero exit code. Code integrations may | ||
| use `validate_research_spec`, `validate_optimization_spec`, or | ||
| `validate_strategy_spec_file` from the lightweight | ||
| `quant_platform_kit.strategy_spec` package. | ||
| Source checkouts may use `python scripts/validate_strategy_spec.py` as an | ||
| equivalent compatibility wrapper. | ||
|
|
||
| ## v1 safety rules | ||
|
|
||
| - Research evaluation requires non-overlapping in-sample/OOS ranges, a locked | ||
| OOS interval, at least three walk-forward folds, PIT and survivorship checks, | ||
| net-of-cost accounting, all-trial recording, and capital/passive/risk-matched/ | ||
| simple-rule benchmarks. | ||
| - Optimization can only vary declared parameters. It requires frozen data, | ||
| universe, benchmark, cost-model, and code references; nested walk-forward; | ||
| a non-reused locked holdout; complete trial recording; and 1x/2x/3x cost | ||
| stress. | ||
| - v1 rejects full Kelly and automatic risk increases. Kelly is retained only | ||
| as a bounded fractional-cap input and human approval remains mandatory. | ||
|
|
||
| `StrategyReviewReport` and `PositionBudgetReport` are intentionally deferred. | ||
| They will consume these immutable inputs rather than duplicate them. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,17 @@ | ||
| #!/usr/bin/env python3 | ||
| """Validate a versioned ResearchSpec or OptimizationSpec JSON artifact.""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import sys | ||
| from pathlib import Path | ||
|
|
||
|
|
||
| ROOT = Path(__file__).resolve().parents[1] | ||
| sys.path.insert(0, str(ROOT / "src")) | ||
|
|
||
| from quant_platform_kit.strategy_spec.cli import main | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| raise SystemExit(main()) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
213 changes: 213 additions & 0 deletions
213
src/quant_platform_kit/schemas/optimization-spec.v1.schema.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,213 @@ | ||
| { | ||
| "$schema": "https://json-schema.org/draft/2020-12/schema", | ||
| "$id": "https://quantplatformkit.local/schemas/optimization-spec.v1.schema.json", | ||
| "title": "OptimizationSpec v1", | ||
| "type": "object", | ||
| "required": [ | ||
| "schema_version", | ||
| "spec_id", | ||
| "research_spec_id", | ||
| "strategy_profile", | ||
| "created_at", | ||
| "frozen_inputs", | ||
| "allowed_parameters", | ||
| "objective", | ||
| "search", | ||
| "validation", | ||
| "stop_rules", | ||
| "promotion" | ||
| ], | ||
| "properties": { | ||
| "schema_version": { "const": "optimization_spec.v1" }, | ||
| "spec_id": { "type": "string", "minLength": 1 }, | ||
| "research_spec_id": { "type": "string", "minLength": 1 }, | ||
| "strategy_profile": { "type": "string", "minLength": 1 }, | ||
| "created_at": { | ||
| "type": "string", | ||
| "format": "date-time", | ||
| "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(\\.[0-9]+)?(Z|[+-][0-9]{2}:[0-9]{2})$" | ||
| }, | ||
| "frozen_inputs": { "$ref": "#/$defs/frozen_inputs" }, | ||
| "allowed_parameters": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "x-qpk-unique-by": "name", | ||
| "items": { "$ref": "#/$defs/parameter" } | ||
| }, | ||
| "objective": { "$ref": "#/$defs/objective" }, | ||
| "search": { "$ref": "#/$defs/search" }, | ||
| "validation": { "$ref": "#/$defs/validation" }, | ||
| "stop_rules": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "promotion": { "$ref": "#/$defs/promotion" } | ||
| }, | ||
| "$defs": { | ||
| "frozen_inputs": { | ||
| "type": "object", | ||
| "required": ["data_manifest_id", "universe_id", "benchmark_ids", "cost_model_id", "code_revision"], | ||
| "properties": { | ||
| "data_manifest_id": { "type": "string", "minLength": 1 }, | ||
| "universe_id": { "type": "string", "minLength": 1 }, | ||
| "benchmark_ids": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "cost_model_id": { "type": "string", "minLength": 1 }, | ||
| "code_revision": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "parameter": { | ||
| "type": "object", | ||
| "required": ["name", "kind"], | ||
| "properties": { | ||
| "name": { "type": "string", "minLength": 1 }, | ||
| "kind": { "enum": ["integer", "number", "choice", "boolean"] }, | ||
| "bounds": { | ||
|
Pigbibi marked this conversation as resolved.
|
||
| "type": "array", | ||
| "minItems": 2, | ||
| "maxItems": 2, | ||
| "items": { "type": "number" } | ||
|
Pigbibi marked this conversation as resolved.
|
||
| }, | ||
| "choices": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "items": { "type": ["string", "number", "boolean"] } | ||
| }, | ||
| "step": { "type": "number", "exclusiveMinimum": 0 } | ||
| }, | ||
| "allOf": [ | ||
| { | ||
| "if": { | ||
| "required": ["kind"], | ||
| "properties": { "kind": { "const": "integer" } } | ||
| }, | ||
| "then": { | ||
| "required": ["bounds"], | ||
| "properties": { | ||
| "bounds": { "items": { "type": "integer" } }, | ||
| "step": { "type": "integer", "exclusiveMinimum": 0 } | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "if": { | ||
| "required": ["kind"], | ||
| "properties": { "kind": { "const": "number" } } | ||
| }, | ||
| "then": { "required": ["bounds"] } | ||
| }, | ||
| { | ||
| "if": { | ||
| "required": ["kind"], | ||
| "properties": { "kind": { "const": "choice" } } | ||
| }, | ||
| "then": { "required": ["choices"] } | ||
| } | ||
| ], | ||
| "additionalProperties": true | ||
| }, | ||
| "objective": { | ||
| "type": "object", | ||
| "required": ["primary_metric", "hard_constraints"], | ||
| "properties": { | ||
| "primary_metric": { "type": "string", "minLength": 1 }, | ||
| "secondary_metrics": { | ||
| "type": "array", | ||
| "items": { "type": "string", "minLength": 1 } | ||
| }, | ||
| "hard_constraints": { | ||
| "type": "array", | ||
| "minItems": 1, | ||
| "items": { "type": "string", "minLength": 1 } | ||
| } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "search": { | ||
| "type": "object", | ||
| "required": ["method", "max_trials", "random_seed"], | ||
| "properties": { | ||
| "method": { "enum": ["grid", "random", "bayesian"] }, | ||
| "max_trials": { "type": "integer", "minimum": 1 }, | ||
| "random_seed": { "type": "integer" } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "validation": { | ||
| "type": "object", | ||
| "required": ["nested_walk_forward", "locked_holdout", "multiple_testing", "cost_stress"], | ||
| "properties": { | ||
| "nested_walk_forward": { | ||
| "type": "object", | ||
| "required": ["enabled", "fold_count", "selection_scope"], | ||
| "properties": { | ||
| "enabled": { "const": true }, | ||
| "fold_count": { "type": "integer", "minimum": 3 }, | ||
| "selection_scope": { "const": "train_validation_only" } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "locked_holdout": { | ||
| "type": "object", | ||
| "required": ["enabled", "reused_for_selection"], | ||
| "properties": { | ||
| "enabled": { "const": true }, | ||
| "reused_for_selection": { "const": false } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "multiple_testing": { | ||
| "type": "object", | ||
| "required": ["method", "trial_ledger_id", "record_all_trials"], | ||
| "properties": { | ||
| "method": { "enum": ["dsr", "pbo", "spa", "reality_check", "fdr", "other_equivalent"] }, | ||
| "trial_ledger_id": { "type": "string", "minLength": 1 }, | ||
| "record_all_trials": { "const": true } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "cost_stress": { | ||
| "type": "object", | ||
| "required": ["multipliers", "required_pass"], | ||
| "properties": { | ||
| "multipliers": { | ||
| "type": "array", | ||
| "minItems": 3, | ||
| "items": { "type": "number", "minimum": 1 }, | ||
| "allOf": [ | ||
| { "contains": { "const": 1 } }, | ||
| { "contains": { "const": 2 } }, | ||
| { "contains": { "const": 3 } } | ||
| ] | ||
| }, | ||
| "required_pass": { "const": true } | ||
| }, | ||
| "additionalProperties": true | ||
| } | ||
| }, | ||
| "additionalProperties": true | ||
| }, | ||
| "promotion": { | ||
| "type": "object", | ||
| "required": [ | ||
| "requires_human_approval", | ||
| "automatic_risk_increase_allowed", | ||
| "full_kelly_allowed", | ||
| "max_fractional_kelly" | ||
| ], | ||
| "properties": { | ||
| "requires_human_approval": { "const": true }, | ||
| "automatic_risk_increase_allowed": { "const": false }, | ||
| "full_kelly_allowed": { "const": false }, | ||
| "max_fractional_kelly": { "type": "number", "exclusiveMinimum": 0, "exclusiveMaximum": 1 } | ||
| }, | ||
| "additionalProperties": true | ||
| } | ||
| }, | ||
| "additionalProperties": true | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.