Skip to content

Introduce Peek: a data-leakage auditor extracted from AgentQuant - #15

Merged
OnePunchMonk merged 1 commit into
mainfrom
feature/peek-leakage-auditor
Jul 23, 2026
Merged

Introduce Peek: a data-leakage auditor extracted from AgentQuant#15
OnePunchMonk merged 1 commit into
mainfrom
feature/peek-leakage-auditor

Conversation

@OnePunchMonk

Copy link
Copy Markdown
Owner

Summary

This is a deliberate pivot of this repo's headline project. AgentQuant (the LLM trading research agent) proved a real, useful thing in its own paper draft: our own walk-forward validation showed the context-aware agent losing to a static baseline. Chasing down why led to auditing our backtest for look-ahead bias — and to the realization that "does my time-series pipeline leak the future into training" is a near-universal ML problem with almost no dedicated tooling (sklearn.TimeSeriesSplit only splits, it doesn't detect leakage).

Peek is that tool, generalized out of AgentQuant's WarmupEnforcer/lookback-guard code into a standalone library:

import peek
report = peek.audit(df, time_col="date", target="y", feature_fn=build_features)
print(report)  # LEAKING / SUSPICIOUS / CLEAN, with proof

Four checks, each gated on what the caller supplies:

  • target_leak (always runs) — flags a feature that's a near-exact copy of the (possibly shifted) target.
  • causality (flagship, needs feature_fn) — recomputes features on a truncated series vs. the full series; if a value changes, the feature saw the future. Catches centered rolling windows, whole-dataset normalization, etc., regardless of shape.
  • split (needs splits/splitter) — train/test temporal overlap, future-dated training rows, missing embargo gap.
  • shuffle (needs pipeline+cv+scorer) — permutation test comparing the real score against scores achievable on randomly shuffled labels.

Also ships a CLI: peek demo (instant leaky-vs-clean walkthrough) and peek audit data.csv --time ... --target ....

What changed

  • New peek/ package (audit orchestrator, report/verdict types, 4 checks, synthetic demo datasets, CLI).
  • 17 new tests (tests/test_peek_*.py) covering every check plus report/verdict logic — all passing, plus all 63 pre-existing AgentQuant tests still pass (80 total).
  • README rewritten to lead with Peek. AgentQuant is fully preserved, untouched in src/, and documented as the origin-story case study in docs/AGENTQUANT.md (moved from the old README content, not deleted).
  • pyproject.toml: added peek console script + package discovery; description updated.
  • CI: added peek/ to the ruff lint step.
  • .gitignore: exclude local .venv/.

Nothing in src/, experiments/, or the original test suite was modified or removed.

Test plan

  • pytest tests/ -v — 80 passed (63 AgentQuant + 17 Peek), no regressions.
  • ruff check peek/ tests/test_peek_*.py — clean (pre-existing lint debt elsewhere in src//tests/ untouched by this PR).
  • python -m peek.cli demo — verified end-to-end: correctly flags both injected leaks (future-copy feature + centered rolling window) on the leaky dataset, and reports CLEAN on the causally-correct dataset built from the same generative process.
  • Reviewer: confirm CI passes on 3.10/3.11/3.12 (dev sandbox only had Python 3.9 available, so package install against the declared >=3.10 requirement wasn't exercised locally — python -m peek.cli and pytest were run directly against source).

Pivot the repo's headline project from AgentQuant (an LLM trading research
agent) to Peek, a focused library that catches look-ahead bias and data
leakage in time-series ML pipelines. Peek generalizes the WarmupEnforcer/
lookback-guard logic built for AgentQuant's backtest engine into a
standalone audit() API with four checks: target_leak (definitive
future-copy detection), causality (truncation-based proof, the flagship
check), split (train/test temporal overlap + embargo), and shuffle
(permutation sanity test on a full pipeline).

AgentQuant is preserved as-is under src/ and documented as the origin
story in docs/AGENTQUANT.md; nothing there was modified or removed.

- Add peek/ package + CLI (`peek demo`, `peek audit`)
- Add 17 tests covering all four checks and report/verdict logic
- Rewrite README to lead with Peek; wire peek console script + packaging
  into pyproject.toml; add peek/ to the CI lint step
@OnePunchMonk
OnePunchMonk merged commit 9edbaa6 into main Jul 23, 2026
4 of 7 checks passed
pull Bot pushed a commit to tianhm/AgentQuant that referenced this pull request Aug 1, 2026
Peek (the data-leakage auditor) has been extracted into its own repo:
https://github.com/OnePunchMonk/peek

This reverts the tree to exactly commit 22bbe0b (the state before
PR OnePunchMonk#15 introduced Peek into this monorepo): removes peek/, its tests,
CLAUDE.md and docs/AGENTQUANT.md (both peek-pivot-specific), and
restores pyproject.toml, README.md, .gitignore, and the CI workflow
to their pre-Peek content.

63 tests passing (the original AgentQuant suite).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant