code-automation

various tests with clause, chatgpt, qwen3, etc

SPARC Portfolio Builder (Configurable Horizon + NYSE/S&P)

Overview

• Daily application that proposes a portfolio over a configurable universe (S&P 500 or NYSE) optimized for the probability of achieving a target return threshold (default +5%) within a configurable horizon (default 5 trading days). Runs at market open (previous close data), proposes adjustments, and publishes daily performance graphs vs SPY.

Key Enhancements

• Universe: S&P 500 (Wikipedia PIT) or full NYSE (cached via NASDAQ Trader list).
• Configurable horizon: --horizon-days for scenario probabilities.
• Portfolio size cap: --max-symbols (daily) and --top-k (backtest).
• Sentiment: rolling z-score sentiment for market and tickers; weighted in the probability model (--sentiment-alpha, --sentiment-beta).
• Probability metrics: estimated P(reach +5% within horizon) and P(lose before ever reaching +5% within horizon).
• Large-universe sampling: --universe-sample to cap fetch size (independent from the target portfolio size cap).
• Target return: --target-return (default 0.05) to set the return threshold used by probability estimates.

Quickstart (Demo Mode, no extra deps)

• Requirements: Python 3.10+

1. Run a daily as-of date
   • python -m spapp.cli daily --date 2025-10-10 --provider demo --universe demo \ --max-symbols 20 --scenarios 1000 --horizon-days 5 --sentiment-window 20
2. Check state and KPIs
   • python -m spapp.cli state
3. Open the daily report
   • reports/daily/2025-10-10/report.html
4. Run a simple backtest (optional)
   • python -m spapp.cli backtest --start 2025-08-01 --end 2025-10-01 \ --provider demo --universe demo --top-k 20
   • Open: reports/backtest/2025-08-01_2025-10-01/report.html

Artifacts

• Snapshots per day: data/snapshots/{YYYY-MM-DD}/ with predictions, weights, and trades.
• Reports: reports/daily/{YYYY-MM-DD}/ with report.html and equity.svg.
• State: state/positions.json, state/performance.json.

Quickstart (Real Data: Yahoo + NYSE)

• Install: pip install yfinance pandas requests lxml
• Fetch and cache NYSE universe: python -m spapp.cli universe --action fetch --source nyse
• Run a daily as-of date with Yahoo + NYSE (example: 10 names, 10-day horizon)
  • python -m spapp.cli daily --date 2025-10-13 --provider yahoo --universe nyse \ --max-symbols 10 --universe-sample 50 --lookback-days 180 \ --scenarios 1000 --horizon-days 10 --target-return 0.05 \ --sentiment-window 20 --sentiment-alpha 0.7 --sentiment-beta 0.5
• Artifacts: data/snapshots/2025-10-13/ and reports/daily/2025-10-13/

Enabling Real Data (Optional)

• Install: pip install yfinance pandas requests lxml

• Use flags:

  • Daily: python -m spapp.cli daily --date 2025-10-10 --provider yahoo --universe nyse --max-symbols 100 --universe-sample 500
  • Backtest: python -m spapp.cli backtest --start 2025-08-01 --end 2025-10-01 --provider yahoo --universe nyse --top-k 100
  • Fetch and cache Wikipedia PIT universe locally:
    • python -m spapp.cli universe --action fetch --source wikipedia
    • Caches: data/cache/wikipedia_sp500_constituents.csv, data/cache/wikipedia_sp500_changes.csv
  • Notes:
    • The Wikipedia fetch requires network and packages: pandas, lxml, requests.
    • After caching, you can use --universe wikipedia offline to reconstruct membership by --date.
    • If cache is missing, the app will prompt you to run the fetch command.

• Fetch and cache NYSE universe locally:

  • python -m spapp.cli universe --action fetch --source nyse
  • Caches: data/cache/nyse_symbols.csv
  • Requires: pandas, requests and network access.

Daily Probability Metrics

• The daily report and snapshot now include scenario-based estimates:
  • prob_reach_target: Probability the portfolio reaches --target-return within --horizon-days trading days.
  • prob_lose_before_target: Probability the portfolio experiences a loss (falls below 0%) before ever reaching --target-return within the horizon.
• These are computed via bootstrap simulation using historical daily returns, sampling correlated daily shocks by drawing whole market days.

Large Universes

• For large universes (e.g., NYSE), use --universe-sample to cap the number of symbols fetched for data (e.g., 500). The optimizer still respects --max-symbols when building the final portfolio.

Testing

• Run integration tests (no extra dependencies):
  • python -m unittest discover -s tests -q
• Included test validates PIT reconstruction using local fixtures.

Yahoo Symbol Normalization

• When using --provider yahoo, symbols with share-class dots are normalized for Yahoo (e.g., BRK.B -> BRK-B). Data are mapped back to the original symbol keys internally.

SPARC Documents

• Situation/Spec: docs/sparc/S.md
• Plan: docs/sparc/P.md
• Action: docs/sparc/A.md
• Results: docs/sparc/R.md
• Conclusion/Check: docs/sparc/C.md

Notes

• The current model is heuristic and dependency-free; replace with trained models and add backtesting when data dependencies are available.

Turnkey Script

• A helper wrapper is available at scripts/daily.sh.
  • Usage: scripts/daily.sh YYYY-MM-DD [provider] [universe]
  • Tunables via env vars: MAX_SYMBOLS, UNIVERSE_SAMPLE, LOOKBACK_DAYS, SCENARIOS, HORIZON_DAYS, TARGET_RETURN, SENT_WIN, SENT_ALPHA, SENT_BETA.
  • Example: MAX_SYMBOLS=10 HORIZON_DAYS=10 TARGET_RETURN=0.05 scripts/daily.sh 2025-10-13 yahoo nyse