Suggestion Robust Montecarlo backtesting

[Gambi97]

Robust Montecarlo backtesting

Hi,
I noticed that a script to verify the robustness of a strategy through Montecarlo methods is not yet present in freqtrade.
The idea of ​​stressing the trading system through a different market situation is a very useful tool to verify the effective efficiency of the strategy used.

What is Montecarlo simulation?

To get a little more into the technician and explain what Montecalro methods are, I'll give you a simple explanation.
The MonteCarlo Method is part of the family of nonparametric statistical methods. It is useful to overcome the computational problems related to exact tests (for example methods based on binomial distribution or calculation, which for large samples generate an excessive number of permutations). The method is used to make estimates through simulations. It is based on an algorithm that generates a series of uncorrelated numbers, which follow the probability distribution that is supposed to have the phenomenon to be investigated. The correlation between numbers is ensured by a chi-square test. " In practice it is possible through the Montecarlo method to generate pseudo-causal values ​​that can be used as series of prices to test the trading system created (or take the prices already analyzed and recombine them in different order).
The results of the trading system applied to the generated series will indicate the robustness of the system. Also in this case we are considering strong hypotheses, we are in fact considering that the average and the variance of future prices is the same as that observed in the past (we have in fact spoken of pseudo-causal values). It is therefore understood that even when a trading system has passed the various robustness tests, you will never be sure that in the future the equity line is in line with that recorded with the historical data in the backtest.

The IDEA

The idea is to create a script that can recombine the candles of a specific period (2 days, 1 month, 1 year; this depends on the timeframe of the strategy) to create create new patterns and verify the behavior of the strategy.
If the strategy is good, the results should not vary as much, if instead the strategy has been hyper-optimized only for a particular market situation then the results will have a high variance.
It is important to verify the variance of the profit, but even more important is to verify the maximum drawdown obtained in the various scenarios.

Conclusion

I hope I have been clear enough and that I have done everything right, I hardly ever use github, if there are questions or curiosities ask me as well.
I should have some documents or links in reference to all this. I will try to find them in the next period.
thank you :)

[xmatthias]

I think the idea of monte carlo simulations is clear to me -
basically you would rerun the same strategy against different datasets.
The outcome will be a confidence interval ... which would tell you that the strategy generated 10% on real backtest data, and with 90% confidence between 12% and 8%.

Example workflow (?)

I think it should be pretty easy to introduce a --monte-carlo-sim flag to backtesting, since i think it makes most sense there ...
so a workflow would be:

• hyperopt the strategy
• transfer found parameters to strategy
• confirm strategy using regular backtesting (Results should be identical to hyperopt -otherwise the parameters were not transferred corectly)
• run backtesting --monte-carlo 1000 - which will take a while as it'll do 1000 backtests with new, unseen data (and one regular backtest, probably - to have the baseline)

In the end, once the "data generation" process is in place, applying this to edge / hyperopt / whatever should be pretty straightforward.

data generation

Now i'm not certain about the data generation you propose.
Wouldn't shuffling candles randomly introduce lookahead bias?
We'd see candles from November in October and viceversa....

Even if you limit the scope of the shuffling (say 2 days) - with crypto pairs, we could have a major jump if the alst candle is used first (assuming a strong trend in one direction over that period of time).

isn't a random walk (randomly generated series) also accomplishing the same thing?
By shuffling the candles we'd be assuming it's a random walk anyway.

I mean, generating randomized timeseries is ~5-10 lines - you randomize returns, add a offset (so it's real prices, not a 0-centered timeseries) - add a random value as high, subtract a random value from the base timeseries as low - and pick a random spot between the 2 as open and close.

I should have some documents or links in reference to all this. I will try to find them in the next period.

Regarding this, i'd be especially interrested in how monte-carlo simulations are usually applied to backtesting ...
I'm looking for confirmation / denial of my assumption made in data generation ...

but quickly searching, i've mainly found cases where the order of trades have been randomized - to identify potential huge drawdown risks and such...

[Gambi97]

hi @xmatthias
the workflow is perfect, I thought so too.
I must start by saying that there are different types of montecarlo applications.
What you are talking about is one of the possible applications, as you have already said well, the idea is to randomize the order of the trades to understand what the maximum drawdown would be in case of "unfortunate" situations.
I personally used this very useful technique to evaluate which money management strategy to use on a trading system (Fixed Ratio, Fixed Fractional Method, Kelly or so on).

What I am referring to is a little more complex.
There are more possibilities to recreate different market situations.
the first is to calculate the variance and average growth of the period and create new values ​​based on those data.
The other even more complicated is to recombine the same candles, however, placing particular conditions.
for example in the world of cryptocurrencies it is not possible that the closing price of a candle does not coincide with the opening price of the next (in short, there must be no jumps).

Both methods are interesting, perhaps that of recombining the trades is a little more truthful and interesting for the crypto world.

However, I found some theoretical books from where I studied money management, the robustness of a system, its repeatability, etc., where the montecarlo method applied to randomizing trades is also explained.
Unfortunately they are paid books and in Italian.
If this can be of interest, I can search for information in English or translate part of it.

in your opinion can it be an interesting addition to freqtrade?

[Gambi97]

I clicked something wrong and closed the issue, sorry I'm not very practical with github for now

[hroff-1902]

I think shuffling will simply break the trends.
And it creates the simulated ohlc data with parameters which cannot be found in the history and cannot be expected in the future.

Assume we have a currency pair with small volatility and smooth trend from 6000 to 8000 during the backtesting period. Shuffling will create strange data with no trend at all, with medium price of 7000 and very high variance of the data -- near candles will vary from 6000 to 8000. Why should we perform backtesting on these artificial data? The result of almost any strategy will be completely different comparing to backtesting on actual historical data and on the actual data on the near real timeranges...

Do you have any paper, written by mathematicians/statisticians, describing this approach (shuffling) applied on time series of data?

[xmatthias]

for example in the world of cryptocurrencies it is not possible that the closing price of a candle does not coincide with the opening price of the next (in short, there must be no jumps).

this is definitely a wrong assumption. Looking at binance (or kraken) charts this is happening very frequently.
consider 1h candles:
last trade was at 10:59:01 for 100 USDT - then orderbook moves (but no order is filled until 10:01:00) - when the order fills at 101 USDT. this will have a jump between open / next close of 1 USD

Very recent example (today):

I clicked something wrong and closed the issue, sorry I'm not very practical with github for now

That'll happen to everyone every now and then - nothing to do with being used to github or not, so don't worry!

I think shuffling will simply break the trends.

I think so too. that's why i'm very doubtful that shuffling will work.
On the other hand - just because LTC/USDT jumped 4% in 1h today will not mean it'll also jump a week from now (or in a month) - even if the preceeding pattern is similar, so the power to predict future from past candles remains questionable overall - so using randomized candlesticks may even provide better results than a "real" historic background, as it'll give you more data - and more data means more situations the strategy has already been through.

Random series

By using a random series - not shuffled, but rebuild from the ground up - using some metrics from the original series (return mean, std, ... ) you can make sure that the strategy will also work in slightly different environments.

Yes, trade results will be different - but that's the whole point - markets change all the time - so a good strategy (based on a good indicator) should ideally also work in different scenarios - or at least it should not fail completely (account blown). This can be accomplished by using good guards / stoplosses - and i think this is where MCS will be able to help.

On the other hand - the Monte Carlo Simulation should also work be using the original candles - and reducing / increasing changes between candles - so instead of a 4% raise, it'll only be a 2% raise (obviously the same would apply in the other direction).

To me big questions

Now the very big question is - what should be randomized / changed for the Monte Carlo Simulations.

• candles?
  • shuffled
  • randomized (build from the ground up)
• trades?
  • this should be fairly simple - https://github.com/gaugau3000/mc_sim_fin from @gaugau3000 seems to implement this method

[Gambi97]

Thinking about your considerations, I agree with you on the lack of consistency of new artificial data sets.
Unfortunately I am unable to find any documentation on the internet for now, indicating that I probably said something stupid or the process is much more complicated than I explained it.

For the moment I would think more about the part of randomizing the order of trades to understand the maximum drawdown (on this topic there are several documents also on the internet).

forgive me if you wasted time looking for information on my first idea.
and thanks for the discussion. ;)

[hroff-1902]

I probably said something stupid

Do not think so, it's a nice discussion issue...

I just would like to see some math (science) behind Monte-Carlo approach and avoid inventing the wheels...

[xmatthias]

forgive me if you wasted time looking for information on my first idea.
and thanks for the discussion. ;)

well no, you didn't - we're discussing - and to me personally, using a (totally randomized) timeseries seems appealing.
That'll prepare a strategy to completely unseen data considering crypto only has backtest data for the last 4-5 years (usually) ... so more data in this context is better.

yes, it should show similar properties, but i think that can be accomplished by carefully designing the candle generator

avoid inventing the wheels...

we wouldn't - it's all virtual - no cars, or other wheel-driven vehicles will be harmed in the process 😆

[hroff-1902]

btw, that's completely same reason why I personally left #2255 hanging... I have not read (recalled, I have had this in the university long time ago) the math. statistics stuff enough to make my conclusion on that yet...

[hroff-1902]

I guess we should invite @gaugau3000 to this discussion as well... 😉

[gaugau3000]

I haven't read the entire topic but what i have implemented in my repo (hope provide more detailed answer after work) is
this. Be aware that what i have implemented work only with independent trades witch may not be the case when you have more than 1 open trades at the same time (BTC fall -> you buy 2 pairs at the same time -> the 2 trades may have close results at the end -> the trade result is correlated (serial correlated) -> the montecarlo analysis consider trades as independents -> the median draw-down calculated is wrong ). Manage 'serial correlation' is in my todo but not yet implemented (a way to simulated this is to regroup trades thats happened at the same time and consider as unique trade when running simulation).

Hope this help have a nice day.

[gaugau3000]

This time i have read all the thread :-)
About the "data generation part" monte carlo simulations work on "independents events" a new candle depend on previous ones not all the time but sometimes this is why TA works (specially in the trends). So you cannot randomized candles. Generate data seems like you have resolve the market that seems pretty difficult.

[acemi1]

Is it possible to do walk forward analysis with freqtrade?
First, walk forvard analysis, then Monte carlo simulation should be done.
Walk forward analysis is considered to be the most reliable solution to Overfitting / Curve fitting problems, which is the most important optimization problem.

https://youtu.be/WF03yaWq_PU?t=4555 ( The video is in Turkish, but there are subtitles.)

[xmatthias]

It will depend on your definition of "walk forward" analysis.
Backtesting (kindof) does this - the only part where it's looking at the full range of data is at the start (basically during the calls to populate_* methods).
The remainder of backtesting is a walk forward analysis already, iterating over time and analyzing what would've happened.

Now it's unclear what you'd "monte carlo simulate" - is it trade sequence? candle patterns?
I'm sorry, but i'll not look a 2h long video in a language i don't understand to get to this - so if you think there's important information in that video, best summarize that for us here.

[Swordfish189]

There is a way to do Montecarlo simulation that would completely avoid the issue of [shuffle vs. random gen candles].
Since it would avoid using candles altogether. It is described in Dr. Van Tharp's "Definitive Guide to Position Sizing",
and it entails first getting the R-multiples distribution of your system. Then instead of simulating the strategy running over different versions of candle tracks, you are actually just randomizing the order of the R series from your total R distribution bank.
In effect it is the shuffling up of the order of the history of the individual trades' P/L. This way enabling the discovery of stuff like large cumulative drawdowns, likeliness of different length streaks, average ending equity, etc...

It does necessitate the 2 following assumptions:

1. You can accurately estimate your systems R-distribution.
2. Your R-distribution will be the same or similar going forward.

But if one were to get a "proper" R-distribution by running a backtest that would include all the main 6 market types:
(bull, bear, sideways) x (calm, volatile)
then the resulting R-distribution could be representative enough for the simulation to have statistical significance.

EDIT:
Just realize this is the same as the above mentioned mc_sim_fin by gaugau3000 🤓

EDIT2:
Except that with R-multiples the simulation would also be informative of the risk to reward ratio. 1R could be abs(stoploss) and then all the P/L would be relative to that.