Could Vibe-Trading submit target weights to a public paper benchmark?

[MooreTechLLC]

Hi Vibe-Trading team,

I’m looking at Vibe-Trading as a possible participant in a public AI investing-agent benchmark.

The benchmark flow is:

• an agent submits target portfolio weights before results are known
• the benchmark platform simulates fills and paper performance
• decisions are logged over time
• performance is compared against SPY and risk-adjusted metrics
• no broker access or live trading is required

Vibe-Trading seems like a strong fit because it already focuses on natural-language finance research, strategy generation, backtesting, and agent workflows.

My question: does Vibe-Trading currently expose a clean way to turn a generated strategy or portfolio analysis into target portfolio weights that could be submitted periodically to an external paper benchmark?

If not, would the right integration path be a small adapter around the strategy/backtest output?

Context: I’m building AllocationAgents.com, a public competition for AI investment agents. I’m looking for serious open-source frameworks that could enter one strategy into the first cohort, but I don’t want to misuse Issues for outreach if this belongs in Discussions.

[warren618]

Thanks @MooreTechLLC — and for asking in Discussions rather than Issues. The paper-only, commit-before-reveal design (no broker, no custody) lines up well with how we think about evaluation, so it's a natural fit.

On the technical question: yes, target portfolio weights are already a first-class output. A strategy in Vibe-Trading is a SignalEngine (generate(data_map) -> {symbol: signal Series}); the backtest normalizes those to portfolio weights (sum(|w|) <= 1) and writes them per-timestamp to a positions.csv artifact. So the last row of that is exactly a set of target weights — no new machinery needed to produce them.

One suggestion on the submission model, in the spirit of a fair benchmark. Having an agent self-submit a weights CSV is the weakest link — it can't be reproduced, and it's hard to rule out cherry-picking or silent drop-out. A more rigorous design is to have the platform run a frozen, reproducible artifact itself: the competitor publishes a pinned signal_engine.py + config.json, and your platform runs the deterministic backtest each period to read the weights. That removes any trust in a self-reported CSV — anyone can re-run the same code on the same data and get the same weights.

The one honest caveat for an LLM-agent framework like ours: the strategy-generation step (the agent doing NL research and writing the SignalEngine) is the non-deterministic part — it needs an LLM and isn't bit-reproducible. But that happens once, up front; the moment it's frozen as a signal_engine.py, weight production downstream is fully deterministic and re-runnable by you, not us. So the reproducible unit is the engine code, not the CSV.

Happy to share a small example signal_engine.py + config.json you could run through that flow to validate it end-to-end. We're not committing to formally enter a cohort right now, but the reproducible-code path above is something we'd be glad to help you design against — it'd make the leaderboard stronger for every framework, not just ours.

[MooreTechLLC]

Thanks, this is really helpful.

The two modes distinction makes sense to me:

1. lightweight external agents submit signed target weights
2. framework-based strategies submit something reproducible like signal_engine.py + config.json

For Vibe-Trading specifically, I’d definitely be interested in seeing a small example of that artifact if you’re willing to share one. No pressure on entering anything formally. This is already useful feedback for how we should design the benchmark side.

[warren618]

Absolutely — here is a tiny reproducible-artifact sketch for the framework-based path.

This is not a formal commitment from us to enter a cohort yet, but it should show the exact shape I had in mind: freeze a signal_engine.py + config.json, let the benchmark runner execute it, then read the final row of artifacts/positions.csv as the target weights.

Directory shape:

agent/runs/allocationagents_demo/
  config.json
  code/
    signal_engine.py

Example code/signal_engine.py:

from typing import Dict

import pandas as pd


class SignalEngine:
    """Minimal causal MA-crossover example.

    generate(data_map) returns {symbol: signal_series}, where:
    1.0 = long, -1.0 = short, 0.0 = flat.
    """

    def __init__(self, fast_window: int = 20, slow_window: int = 60):
        self.fast_window = fast_window
        self.slow_window = slow_window

    def generate(self, data_map: Dict[str, pd.DataFrame]) -> Dict[str, pd.Series]:
        signals = {}

        for symbol, df in data_map.items():
            close = df["close"].astype(float)

            fast = close.rolling(self.fast_window, min_periods=self.fast_window).mean()
            slow = close.rolling(self.slow_window, min_periods=self.slow_window).mean()

            signal = pd.Series(0.0, index=df.index)
            signal[fast > slow] = 1.0
            signal[fast < slow] = -1.0

            # Use only information available after the previous completed bar.
            signals[symbol] = signal.shift(1).fillna(0.0)

        return signals

Example config.json:

{
  "source": "yfinance",
  "codes": ["AAPL.US", "MSFT.US", "SPY.US"],
  "start_date": "2023-01-01",
  "end_date": "2024-12-31",
  "interval": "1D",
  "engine": "daily",
  "initial_cash": 100000,
  "commission": 0.001,
  "extra_fields": null
}

Run it from the repo checkout with:

cd agent
python -m backtest.runner runs/allocationagents_demo

The runner writes:

agent/runs/allocationagents_demo/artifacts/positions.csv
agent/runs/allocationagents_demo/artifacts/metrics.csv
agent/runs/allocationagents_demo/run_card.json
agent/runs/allocationagents_demo/run_card.md

For your benchmark, the integration point would be:

1. clone/pin the artifact,
2. run the backtest over the benchmark period,
3. read the last timestamp row from artifacts/positions.csv,
4. submit that row as the target weights.

The run card is useful for auditability because it records hashes for the config and strategy file. So the cleanest benchmark contract is probably: “submit a frozen strategy artifact, not a self-reported weights CSV.”

Happy to help refine the artifact shape if AllocationAgents wants a stricter schema around rebalance dates, max gross exposure, long-only vs long/short, or benchmark-owned data.