feat(analysis): deterministic outcome evaluation (mureo_outcome_evaluate)#361
Merged
Conversation
…ate) Turn the observation-window review (daily-check) and /learn signal from an LLM judgement call into a reproducible verdict. New pure engine mureo/analysis/outcome_eval.py compares a before/after metric snapshot and returns improved / regressed / inconclusive per metric + overall: - Metric direction is explicit (cpa/cpc/cpl/cpm lower-is-better; conversions/ctr/cvr/roas higher-is-better; cost/spend/clicks/impressions volume-only, never scored). - A change within +/-noise_pct (default 10%) or a zero/absent baseline is inconclusive — no fabricated swing (same noise discipline as the anomaly detector). - Pure and platform-agnostic: works for any platform (google_ads / meta_ads / tiktok_ads / plugins) as long as the caller feeds comparable metric names. Exposed as the mureo_outcome_evaluate MCP tool (before = action_log metrics_at_action, after = current). daily-check's evidence step now calls it instead of eyeballing, and normalizes hosted/plugin metric names to the standard keys first so the verdict works for TikTok too. TDD: tests/test_outcome_eval.py (pure eval + MCP tool). Tool-count assertions updated (mureo context 8->9, total 189->190).
This was referenced Jul 7, 2026
Merged
hyoshi
added a commit
that referenced
this pull request
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Feature 2 of the OSS roadmap. Turns the observation-window review (daily-check) and the
/learnsignal from an LLM judgement call into a reproducible verdict.mureo/analysis/outcome_eval.py:evaluate_metric/evaluate_outcomecompare a before/after metric snapshot → improved / regressed / inconclusive per metric + overall. Metric direction is explicit (cpa/cpc/cpl/cpm lower-is-better; conversions/ctr/cvr/roas higher-is-better; cost/spend/clicks/impressions volume-only, never scored). A change within ±noise_pct(default 10%) or a zero/absent baseline is inconclusive — no fabricated swing.mureo_outcome_evaluate(before = action_logmetrics_at_action, after = current numbers).Test plan
tests/test_outcome_eval.py— pure eval (directions, noise band, zero baseline, string coercion) + MCP tool registration/dispatchruff/black/mypyclean onmureo/(244 files)Roadmap: #1 policy gate (merged #360) · #2 outcome eval (this) · #3 generic STATE.json anomaly (pending). Calculation-tier features (#2, #3) reach hosted MCPs; hard enforcement for those needs the gateway (#359).
https://claude.ai/code/session_0195WBK6YopTveq9RAQE2xMz