First of all, excellent work on TradingAgents.
The architecture, agent coordination design, and historical data pipeline are very impressive.
However, I would like to raise a concern regarding a potential evaluation issue in LLM-based backtesting:
Temporal Knowledge Leakage from the Base LLM
From the documentation and implementation, TradingAgents already appears to handle traditional future leakage correctly:
- date-aware data fetching
- historical news restriction
- timestamp-based retrieval
- historical OHLCV snapshots
This prevents explicit future data exposure.
However, there is another layer of leakage that may still exist:
Model-Level Future Knowledge Contamination
For example:
- using a 2026-trained LLM
- while backtesting decisions in 2023
Even if the prompt context is historically constrained, the model weights themselves may already encode:
- macro outcomes
- company trajectories
- major crises
- sector bubbles
- future market narratives
Examples:
- NVDA AI boom
- FTX collapse
- SVB failure
- post-rate-hike market behavior
- inflation outcomes
This means the model may unintentionally perform:
- hindsight reasoning
- retroactive narrative fitting
- future-aware macro interpretation
rather than genuine ex-ante reasoning.
Question
Has the team considered this issue formally?
Specifically:
-
Do you view TradingAgents backtests as:
- historically grounded simulations
- or strictly causal backtests?
-
Have you explored:
- time-frozen checkpoints
- older foundation models
- cutoff-constrained continual pretraining
- temporal contamination evaluation
-
Are there plans to introduce:
- model cutoff declarations
- leakage benchmarks
- temporal purity evaluation protocols
Possible Research Direction
A potentially rigorous protocol could be:
| Backtest Period |
Model Knowledge Cutoff |
| 2023 |
<= 2022 |
| 2024 |
<= 2023 |
with:
- retrieval cutoff
- embedding cutoff
- reranker cutoff
- memory cutoff
all aligned temporally.
Beside above questions, what solutions has your team come up with for this problem?
I believe this is becoming an important issue for the entire LLM-for-finance research community, especially as reasoning models become stronger.
Would love to hear the team's perspective on this.
First of all, excellent work on TradingAgents.
The architecture, agent coordination design, and historical data pipeline are very impressive.
However, I would like to raise a concern regarding a potential evaluation issue in LLM-based backtesting:
Temporal Knowledge Leakage from the Base LLM
From the documentation and implementation, TradingAgents already appears to handle traditional future leakage correctly:
This prevents explicit future data exposure.
However, there is another layer of leakage that may still exist:
Model-Level Future Knowledge Contamination
For example:
Even if the prompt context is historically constrained, the model weights themselves may already encode:
Examples:
This means the model may unintentionally perform:
rather than genuine ex-ante reasoning.
Question
Has the team considered this issue formally?
Specifically:
Do you view TradingAgents backtests as:
Have you explored:
Are there plans to introduce:
Possible Research Direction
A potentially rigorous protocol could be:
with:
all aligned temporally.
Beside above questions, what solutions has your team come up with for this problem?
I believe this is becoming an important issue for the entire LLM-for-finance research community, especially as reasoning models become stronger.
Would love to hear the team's perspective on this.