Introduce Peek: a data-leakage auditor extracted from AgentQuant - #15
Merged
Conversation
Pivot the repo's headline project from AgentQuant (an LLM trading research agent) to Peek, a focused library that catches look-ahead bias and data leakage in time-series ML pipelines. Peek generalizes the WarmupEnforcer/ lookback-guard logic built for AgentQuant's backtest engine into a standalone audit() API with four checks: target_leak (definitive future-copy detection), causality (truncation-based proof, the flagship check), split (train/test temporal overlap + embargo), and shuffle (permutation sanity test on a full pipeline). AgentQuant is preserved as-is under src/ and documented as the origin story in docs/AGENTQUANT.md; nothing there was modified or removed. - Add peek/ package + CLI (`peek demo`, `peek audit`) - Add 17 tests covering all four checks and report/verdict logic - Rewrite README to lead with Peek; wire peek console script + packaging into pyproject.toml; add peek/ to the CI lint step
2 tasks
pull Bot
pushed a commit
to tianhm/AgentQuant
that referenced
this pull request
Aug 1, 2026
Peek (the data-leakage auditor) has been extracted into its own repo: https://github.com/OnePunchMonk/peek This reverts the tree to exactly commit 22bbe0b (the state before PR OnePunchMonk#15 introduced Peek into this monorepo): removes peek/, its tests, CLAUDE.md and docs/AGENTQUANT.md (both peek-pivot-specific), and restores pyproject.toml, README.md, .gitignore, and the CI workflow to their pre-Peek content. 63 tests passing (the original AgentQuant suite).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is a deliberate pivot of this repo's headline project. AgentQuant (the LLM trading research agent) proved a real, useful thing in its own paper draft: our own walk-forward validation showed the context-aware agent losing to a static baseline. Chasing down why led to auditing our backtest for look-ahead bias — and to the realization that "does my time-series pipeline leak the future into training" is a near-universal ML problem with almost no dedicated tooling (
sklearn.TimeSeriesSplitonly splits, it doesn't detect leakage).Peek is that tool, generalized out of AgentQuant's
WarmupEnforcer/lookback-guard code into a standalone library:Four checks, each gated on what the caller supplies:
feature_fn) — recomputes features on a truncated series vs. the full series; if a value changes, the feature saw the future. Catches centered rolling windows, whole-dataset normalization, etc., regardless of shape.splits/splitter) — train/test temporal overlap, future-dated training rows, missing embargo gap.pipeline+cv+scorer) — permutation test comparing the real score against scores achievable on randomly shuffled labels.Also ships a CLI:
peek demo(instant leaky-vs-clean walkthrough) andpeek audit data.csv --time ... --target ....What changed
peek/package (audit orchestrator, report/verdict types, 4 checks, synthetic demo datasets, CLI).tests/test_peek_*.py) covering every check plus report/verdict logic — all passing, plus all 63 pre-existing AgentQuant tests still pass (80 total).src/, and documented as the origin-story case study indocs/AGENTQUANT.md(moved from the old README content, not deleted).pyproject.toml: addedpeekconsole script + package discovery; description updated.peek/to the ruff lint step..gitignore: exclude local.venv/.Nothing in
src/,experiments/, or the original test suite was modified or removed.Test plan
pytest tests/ -v— 80 passed (63 AgentQuant + 17 Peek), no regressions.ruff check peek/ tests/test_peek_*.py— clean (pre-existing lint debt elsewhere insrc//tests/untouched by this PR).python -m peek.cli demo— verified end-to-end: correctly flags both injected leaks (future-copy feature + centered rolling window) on the leaky dataset, and reports CLEAN on the causally-correct dataset built from the same generative process.>=3.10requirement wasn't exercised locally —python -m peek.cliand pytest were run directly against source).