V1 toggle specialist#126
Merged
Merged
Conversation
Adds a weeks-based campaign-length grid alongside the existing months grid, and a study script that regression-tests the two toggle-capable methods (toggle_specialist, power_model) on Hill of Towie. The weeks grid is additive: StudyConfig takes exactly one of campaign_months / campaign_weeks, and leaderboard/plot_campaign_curves take a length_col defaulting to campaign_months. Every existing months-based call is therefore bit-identical, so the committed power_model baseline stays valid. months-only accessors (CampaignWindow.months, StudyConfig.max_activity_months) raise on a weeks study rather than silently reporting weeks as months. study_toggle_methods_compare.py scores a placebo plus a symmetric +/-2% Cp pair over 1/2/4/8 weeks -- the small-signal, short-campaign regime a real toggle campaign lives in. It reports raw deltas against a committed benchmark: ground truth is deterministic in (config, seed), so an unchanged method must diff to exactly 0.0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new toggle-only baseline method (toggle_specialist) plus a dedicated short-campaign (weeks-grid) regression harness, while generalizing the harness to support campaign-length grids in either months or weeks and standardizing conditional-axis selection/validation across methods.
Changes:
- Introduce
ToggleSpecialistMethod(toggle-only energy-ratio baseline) with optional per-power-bin reporting and rich diagnostics. - Generalize campaign windowing/scoring/plotting/leaderboards to support a weeks-based campaign grid in addition to months.
- Replace
PowerModelMethod.conditional_upliftwithconditions=...+ sharedvalidate_conditions, and add a new toggle-methods benchmark script + committed baseline.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/benchmarking/harness/test_replicates.py | Adds coverage for weeks-grid studies and campaign-grid validation on StudyConfig. |
| tests/benchmarking/harness/test_conditions.py | Adds unit tests for shared validate_conditions. |
| tests/benchmarking/harness/test_campaign.py | Adds weeks-grid campaign-window tests and grid-validation tests. |
| tests/benchmarking/baselines/test_toggle_specialist.py | Comprehensive unit tests for the new ToggleSpecialistMethod behavior and diagnostics. |
| tests/benchmarking/baselines/test_study_toggle_methods_compare.py | Tests the new toggle-methods regression harness behavior (selection, leaderboard, diffing, plots). |
| tests/benchmarking/baselines/test_study_power_model_compare.py | Updates expectations to the new conditions API. |
| tests/benchmarking/baselines/test_power_model_method.py | Migrates tests from conditional_uplift to conditions and adds conditions-selection tests. |
| tests/benchmarking/baselines/test_naive_ratio.py | Minor formatting-only adjustments. |
| docs/v1/findings.md | Documents the new toggle-specialist per-bin estimator and the new weeks-based regression harness. |
| benchmarking/synthetic/ground_truth.py | Clarifies comments re: baseline/upgrade invariance for power binning. |
| benchmarking/harness/scoring.py | Records campaign length under the correct column (campaign_months vs campaign_weeks). |
| benchmarking/harness/replicates.py | Extends StudyConfig to accept exactly one of months/weeks grids; adds generic accessors. |
| benchmarking/harness/plots.py | Generalizes campaign-curve plotting to a configurable campaign-length column. |
| benchmarking/harness/leaderboard.py | Generalizes leaderboard grouping keys via a configurable campaign-length column. |
| benchmarking/harness/conditions.py | Adds shared validate_conditions and updates power-bin comment wording. |
| benchmarking/harness/campaign.py | Adds weeks-grid support, CampaignUnit, generic window length fields, and grid resolution/validation. |
| benchmarking/diagnostics/density.py | Updates module docstring text (comment-only change). |
| benchmarking/baselines/toggle_specialist.py | New toggle-only baseline method implementation + diagnostics and optional per-bin reporting. |
| benchmarking/baselines/study_toggle_methods_compare.py | New weeks-grid regression harness for toggle methods with benchmark record/diff workflow. |
| benchmarking/baselines/study_toggle_methods_compare_baseline.json | Adds committed benchmark baseline cells for the new harness. |
| benchmarking/baselines/study_power_model_compare.py | Updates baseline provenance to use campaign_lengths accessor. |
| benchmarking/baselines/power_model/method.py | Replaces conditional_uplift with conditions, adds validation, and updates conditional power bin labeling. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
* toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * toggle specialist uncertainty WIP * clean up baseline loose end * address PR comments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new toggle specialist model and study focused on short toggle campaigns