Benchmark results and public-data reproductions

[initial-d]

This is the public thread for sharing benchmark results, public-data reproductions, and setup feedback for ml-quant-trading.

What would be most useful

• CPU/GPU benchmark results for tensor factor primitives.
• Public-data reproductions using yfinance, Baostock, or another accessible data source.
• Factor-engine edge cases around missing data, masks, halts, limit-up / limit-down events, and cross-sectional ranking.
• Notebook or setup issues that make the project harder to try.

Useful links

• Repository: https://github.com/initial-d/ml-quant-trading
• v0.1.0 release: https://github.com/initial-d/ml-quant-trading/releases/tag/v0.1.0
• Paper: https://arxiv.org/abs/2507.07107
• Benchmark issue: Call for benchmarks and public-data reproductions #12
• Community outreach notes: https://github.com/initial-d/ml-quant-trading/blob/main/docs/community_outreach.md

Suggested benchmark report shape

Hardware:
OS:
Python:
PyTorch:
Command:
Panel size:
Device:
Runtime:
Notes:

This project is research software, not financial advice or a live trading recommendation. Please avoid posting proprietary datasets, secrets, or broker/account information.

[initial-d]

Update: the first maintainer CPU benchmark is now on the benchmark board:

https://github.com/initial-d/ml-quant-trading/blob/main/docs/benchmark_board.md

Current baseline: MacBook Air / Apple M5 / macOS / PyTorch 2.8.0 / CPU-only. More useful comparisons: CUDA GPU, Linux CPU, larger panels, and public-data reproduction notes.

[initial-d]

Update: a public-data mini reproduction note is now available:

https://github.com/initial-d/ml-quant-trading/blob/main/docs/public_data_mini_reproduction.md

It uses yfinance, 10 liquid US equities, a six-factor subset, and one-day forward rank IC. More useful next reports: larger public universes, ETFs, A-share public-provider runs, and backtest variants with explicit costs.

[initial-d]

Maintainer update: added a larger public-data validation benchmark in 9cc65cd.

New docs:

• Public-data validation benchmark: https://github.com/initial-d/ml-quant-trading/blob/main/docs/public_data_validation.md
• Reality Check and Validation Status: https://github.com/initial-d/ml-quant-trading/blob/main/docs/reality_check.md

What changed:

• Added scripts/public_data_validation.py with yfinance presets for 100 US large caps, ETF-50, and mixed stock/ETF universes.
• Added walk-forward validation with 504-day train / 63-day test / 63-day step defaults.
• Added transaction cost + slippage reporting, turnover, drawdown, gross/net metrics, and active metrics versus equal weight.
• Added baseline comparisons: equal weight, 20-day momentum, Alpha101 mean score, MLP on Alpha101 features, Transformer on Alpha101 features.
• Recorded a 100-stock yfinance reference run from 2021-01-01 to 2025-01-01.

Important result from the maintainer run: equal weight was the strongest net baseline in this public-data reference run. That is useful negative evidence and a reminder that this benchmark is a validation diagnostic, not a claim of deployable alpha.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28733020426

Community benchmark runs on different universes, GPUs, and data providers are very welcome.

[initial-d]

Maintainer update: validation reports are now easier to share.

Commit: 4b39b7b

python scripts/public_data_validation.py ... now writes:

• summary.md for human-readable results
• summary.csv for spreadsheets
• summary.json for future aggregation scripts
• metadata.json with command, environment, CUDA, data coverage, and config
• submission.md as a copy-ready GitHub issue body

I also added a dedicated Public-data validation report issue template so community runs can include the same command, environment, coverage, and result fields.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28733879832

[initial-d]

Maintainer update: added validation report aggregation.

Commit: 507fae1

New helper:

python scripts/aggregate_validation_reports.py artifacts/public_data_validation

It scans one or many summary.json files from public-data validation runs and writes a compact Markdown/CSV leaderboard with source, preset, date range, panel size, strategy, return, Sharpe, drawdown, turnover, coverage, and environment fields.

This should make future community reports easier to compare without treating the table as a trading-performance claim.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28734048384

[initial-d]

Maintainer update: added a validation report audit gate.

Commit: dd2c151

New helper:

python scripts/audit_validation_report.py artifacts/public_data_validation/summary.json

The audit checks missing metadata, missing result rows, low data coverage, tickers with no data, missing equal-weight baseline, non-finite metrics, negative cost settings, unusual turnover/drawdown, and non-positive final equity.

This is meant to improve reproducibility hygiene for community reports. It is not a claim that any strategy is profitable.

CI is green on Python 3.9, 3.10, and 3.11: https://github.com/initial-d/ml-quant-trading/actions/runs/28734249377

[initial-d]

Small update for reproducibility reports: the public-data validation harness now supports effective-cost sensitivity grids.

Example:

python scripts/public_data_validation.py \
  --source yfinance \
  --preset us-large-100 \
  --cost-grid-bps 0,7,15,30

This writes cost_sensitivity.md, cost_sensitivity.csv, and cost_sensitivity.json, and also embeds the rows in summary.json under cost_sensitivity. The intent is to make reports easier to compare when turnover-heavy strategies only work under one narrow cost/slippage assumption.

CI is passing on Python 3.9, 3.10, and 3.11 for commit 3549ce9.