I spent two months building an automated trading system. It was losing money. The backtest said it was making money.
Both were correct, which is the problem.
Every bug below produced a better result than the truth. None of them threw an error, and none of them looked like bugs — they looked like a strategy that worked. I found them in a single day of going back over the validation pipeline, and each one, on its own, was enough to invert the answer.
If you're building anything where a number decides what you do next, these are worth knowing. They're not exotic. Three of the four are variations on the same mistake.
The system trades daily temperature markets. Each market closes at a fixed time — for these, 12:00 UTC. After that no order can be placed.
My backtest walked forward through each day and scored an entry at whatever the price was when the signal fired. It never checked whether the market was still open.
So it was buying at prices that existed only after the book had shut. And those prices are systematically better, because by then the outcome is largely known and the market has converged. The backtest was buying certainty at uncertainty's price.
How it hid: everything about it looks correct. There is a price, there is a timestamp, the arithmetic is right. Nothing errors. The only tell is that the returns are too good, and "too good" is exactly the thing you don't investigate when you want it to be true.
The fix was one clamp: never score an entry at or after the market's own close time. That single line removed most of the apparent edge.
if as_of >= market_close_utc:
break # not tradeable, not scoreableFixing bug 1 required widening the scan window. Which introduced bug 2.
The system watches 29 cities. A market for "the high in Chicago on August 3rd" resolves against Chicago's local day — but the trading session runs on UTC. For a city at UTC−5, scanning from 00:00 UTC starts at 19:00 the previous local evening.
So the strategy would look at yesterday's completed afternoon, see a high temperature that had clearly peaked, and confidently place a bet on today's market using it.
It won often enough to look fine, because tomorrow's weather correlates with today's.
How it hid: it produced more trades and a plausible win rate. More signals feels like a working system, not a broken one.
The fix was a cross-date gate — the observations and the market must belong to the same local day. When I added it, one population of cities went from 38 trades to 1. Thirty-seven of thirty-eight had been wrong-day bets.
Same family, one level down.
The rule depends on the day's maximum temperature so far. My function took every observation up to the current moment and returned the maximum.
Every observation it had fetched. Which included the previous day.
The tell was a single line in a loss report: a high of 101°F at local midnight. A daily maximum cannot occur at midnight. It was yesterday's afternoon peak, still sitting in the window.
The live system had this right — it filtered observations to the local day. The backtest didn't. The two had drifted apart, and only the backtest was making claims.
The fix was filtering observations to the same local date as the market. Three lines.
The lesson that generalises: if your validation code and your production code both compute the same concept, they will drift. Make them share the function or make them prove they agree. Mine differed by a filter that existed in one and not the other, and the one without it was the one generating the numbers I trusted.
This is the interesting one, because it isn't a coding error at all.
I was testing an exit rule: buy, then sell at the first price 50% above entry. Measured over 509 trades it returned +0.153 per dollar with a t-statistic of +4.89 — which, if you take t-statistics at face value, is overwhelming.
The same 509 trades, held to resolution instead, returned +0.017.
Identical entries. Different scoring. The gap is the entire result.
The rule says sell at the first trade at or above +50%. But prices jump. If a market moves from 30¢ to 55¢, the first observed trade above target isn't at +50%, it's at +83% — and the rule books that overshoot as profit. Meanwhile losers keep their full loss, because there's no equivalent overshoot on the way down when you're holding to zero.
15% of entries were booked at over +60%.
The t-statistic was real. It was measuring a real, consistent, systematic effect. The effect was my measurement rule, not the market.
This is the one worth internalising: a large sample makes systematic bias look more significant, not less. With random noise, more data converges on the truth. With a biased estimator, more data converges confidently on the wrong answer. A t of 4.89 told me the bias was consistent.
The fix wasn't code. It was reporting the hold-to-resolution number beside every exit-rule number, always. When they disagree, the difference is your measurement, not your edge.
I now assume any result I like is wrong until it survives:
- A close/availability check. Could this transaction have happened, at that moment, at that price?
- A same-entity check. Do the data and the thing being predicted refer to the same day, the same object, the same scope?
- A shared-implementation check. Does production compute this the same way the test does? Prove it, don't assume it.
- A null-scoring check. What does the simplest possible scoring rule say about these exact same events? If a clever metric disagrees with the naive one, the burden is on the clever one.
None of these are sophisticated. The reason they matter is that all four bugs were silent and flattering — the failure mode that survives longest is the one that tells you what you want to hear.
After fixing all four, the corrected result was still positive but far smaller, and it then failed out-of-sample: an apparent edge of +0.0606 in the first half of the data became +0.0018 in the second.
The strategy didn't work. Two months of building, and the honest answer was no.
That's a better outcome than it sounds. It cost paper money and a weekend of statistics rather than a funded account and six months. The bugs were the expensive part — not because of what they cost, but because of how long they'd have let me keep going.
Related: unattended — a lock that holds under a race, and a liveness check a running PID can't fool. Also written after the obvious version failed.