Enhanced Backtesting Tables

[yazeed]

Updated Strategy Summary to be consistent with other backtesting tables...

=========================================================== STRATEGY SUMMARY ===========================================================
| Strategy    |   Buy Count |   Avg Profit % |   Cum Profit % |   Tot Profit BTC |   Tot Profit % | Avg Duration   |     Wins | Losses |
|:------------|------------:|---------------:|---------------:|-----------------:|---------------:|:---------------|---------:|-------:|
| Strategy1   |         429 |           0.36 |         152.41 |       0.00762792 |          76.20 | 4:12:00        |      186 |    243 |
| Strategy2   |        1487 |          -0.13 |        -197.58 |      -0.00988917 |         -98.79 | 4:43:00        |      662 |    825 |

[coveralls]

Coverage remained the same at 97.194% when pulling 301f8a2 on yazeed:strategy_summary_mods into b5ee4f1 on freqtrade:develop.

[yazeed]

More work to be done here.

[yazeed]

I've worked all the way through to modifying the strategy table to calculating expectancy via the formula:

Expectancy = (Probability of Win * Average Absolute Win) - (Probability of Loss * Average Absolute Loss) / stake_amount

divided by stake_amount, to keep the final value consistent regardless of the stake_amount used in backtesting.

My primary finding after adding all of these columns is that it adds little extra insight, but thought some may find it useful nonetheless.

Updated tables in screenshots above are not committed yet, and I'm not sure if you'd like that many information in the strategy results table, some could find them useful. I noticed Avg Profit in Stake_currency was also missing, so if you agree to the guideline in the screenshot above, I could add in that as well.

I'd like to hear your thoughts.

[yazeed]

By the way, here's an additional iteration with a draws column which I figured was also missing, everywhere not just in the strategy results table, since if you're using minimal_roi to sell at 0% profit after a certain time, there will be a lot of these draws and it's important to see them, not committed yet, just want to see your thoughts here first.

Notice, how without the breakdown of wins/draws/losses you would think your strategy is winning a lot more than you think.

[xmatthias]

I don't mind more information in general ... but i'm a bit concerned that too much information can be overwelming.

Also, screen width can be a problem too (not sure how many chars in width this takes ... but i know one thing - once this starts wrapping lines - it'll loose usefulness.

maybe (just an idea) we can not show most of these new columns for all pairs - and only show them as a summary at the end?

---

Also, expectancy will require good explanation - what's a good / bad expectancy value?
We've seen it with edge - 100ds of questions of "this is my result - is it good?

Assuming the calculation is the same than in edge - i find it weired that you have 95% win rate (JollingerBands), an average profit of 0.19%, total profit of 105% - but still only expect your total capital to expand by 0.39% (0.0039) ...

Maybe i'm misreading this (i've not done much with expectancy before) - but should't this be aligned to the result of total-profit (somehow) ?

[yazeed]

Sorry for the repetitive posts, just documenting my journey step by step, here's what the final table looks like (not committed to this PR yet)

=========================================================================================================== STRATEGY SUMMARY ===========================================================================================================
| Strategy            |   Buy Count |   Avg Profit % |   Cum Profit % |   Avg Profit BTC |   Tot Profit BTC |   Tot Profit % | Avg Duration   | Wins         | Draws     | Losses      |   Avg Win BTC |   Avg Loss BTC |   Expectancy |
|:--------------------|------------:|---------------:|---------------:|-----------------:|-----------------:|---------------:|:---------------|:-------------|:----------|:------------|--------------:|---------------:|-------------:|
| JollingerBands4h165 |         129 |           0.55 |          71.37 |       0.00027689 |       0.03571871 |           7.14 | 18:59:00       | 90 (69.77%)  | 7 (5.43%) | 32 (24.81%) |   0.00108538  |    -0.00193643 |   0.00553778 |
| JollingerBands4h169 |         205 |           0.13 |          26.94 |       0.00006577 |       0.01348271 |           2.69 | 8:00:00        | 121 (59.02%) | 0 (0.0%)  | 84 (40.98%) |   0.000853135 |    -0.00106841 |   0.00131539 |
| JollingerBands4h181 |         102 |           1.10 |         111.80 |       0.00054858 |       0.05595538 |          11.18 | 1 day, 9:11:00 | 80 (78.43%)  | 5 (4.9%)  | 17 (16.67%) |   0.00130815  |    -0.0028645  |   0.0109716  |
| JollingerBands4h351 |         192 |           0.24 |          45.76 |       0.00011928 |       0.02290264 |           4.58 | 6:08:00        | 122 (63.54%) | 0 (0.0%)  | 70 (36.46%) |   0.000703305 |    -0.00089858 |   0.00238569 |
| JollingerBands4h411 |         224 |           0.15 |          34.64 |       0.00007739 |       0.01733470 |           3.46 | 6:29:00        | 153 (68.3%)  | 0 (0.0%)  | 71 (31.7%)  |   0.000633889 |    -0.00112184 |   0.00154774 |

[hroff-1902]

what if I do not have such a wide display?

[yazeed]

what if I do not have such a wide display?

Then you have a problem, which you can solve by scrolling to the right. I was just thinking about that.

[xmatthias]

This final table is definitely too wide.
It'll wrap on most pc's with full-hd resolution ...

Then you have a problem, which you can solve by scrolling to the right. I was just thinking about that.

No you can't - a regular terminal will wrap the lines - so you'll have additional new-lines within the table - which will make the result unreadable.

[yazeed]

What do you suggest we can do about it? Not sure it would look better if I break it onto two tables.

It has all the info anyone might need to analyze a strategy, which is in my own rationale, the purpose of that strategy comparison table, to be able to thoroughly and delicately compare between strategies.

[xmatthias]

an (extreme) example from my terminal (i've intentionally reduced the wide as i don't have the new columns available).

Now the information is still okish as only the last column wraps ... but you see the wrapping.
Once it starts wrapping, its' very difficult to get anything usefull out of this - so instead of giving users all information needed, you've removed most information from being easily accessible.

A possible alternative:
All necessary / available information should be put to the json file - and code to load the file to a dataframe (we got that) is added to the jupyter notebook template - for further analysis (if more info is available).

So everyone can open an interactive session (no notebook required!), load the file - and display the columns he's interrested in - which i assume will differ from person to person.

[yazeed]

an (extreme) example from my terminal (i've intentionally reduced the wide as i don't have the new columns available).

Now the information is still okish as only the last column ... but you see the wrapping.
Once it starts wrapping, its' very difficult to get anything useful out of this - so instead of giving users all information needed, you've removed most information from being easily accessible.

I understand, I'm not saying this is okay, I actually prefer the 133 guideline provided in other tables, but that kinda limits information available. I'm not sure which approach should we take to fix this.

Maybe separate information into two tables?

[hroff-1902]

Printing extra options are just fine... But output quickly becomes too wide for many displays and screens... That's why I did not add many interesting metrics yet...

I guess it should be controlled by the user, in cli options, for example. Like indicator* options for plotting. So if you need extra metrics printed, use

$ freqtrade backtesting ... --extra-metrics Expectancy Draws SharpeRatio CalmarRatio

or something like that...

The user will then be able to choose what to print depending on her screen...

[xmatthias]

I think I do have a better idea / proposal (came to me while driving).

We could use a HTML output for this - this would not have these limitations, and would give us a lot more flexibility.

It doesn't have to be anything fancy - a jinja2 template (look at our template folder), rendered with the output as a table (<table>) (multiple tables).

In HTML / a browser, you can pan (scroll left / right) with easy / without limitation.
Basically, we can rebuild the table output as it is here - but with the additional columns (to start with).

Just render the template, storing it as a file in userdata/backtest_results/ - ideally as results<timestamp)>.html - and open the users default browser (optionally??).

In the end, we can add more things to it as we have several open issues about added metrics and the highest pain-point was the "how to display".

It'll also tie in rather nicely with the (long-term) plan to have an UI - which will require this step anyway - so we can also start building that step by step.

---

for this PR, I'd like to

• Merge this PR as is (no additional columns - just "stylistic fixes")
• move the discussion around the html template to an issue / a seperate PR which aims at introducing HTML backtesting results.

[yazeed]

I like the HTML idea. I could work on that in a different PR.

Feel free to merge this PR as it stands, I haven't made any of the expanded table commits, it's all local on my side, as I was merely on a quest exploring what's missing in the strategy results table.

[hroff-1902]

yeah, html is cool. 👍

[xmatthias]

Ok, will merge this as is. 👍

Please make sure to keep the code for how to generate the extended table - you'll need the calculations for the HTML part anyway. 😄

Thanks for offering to work on the HTML part, too - i think that'll be a GREAT new feature, so i'm really looking forward to that!

Feel free to open an "early"/WIP PR so we can collaborate if you need any help or got questions regarding the templating (not sure if you've used that before).

[yazeed]

Ok, will merge this as is. 👍

Please make sure to keep the code for how to generate the extended table - you'll need the calculations for the HTML part anyway. 😄

Thanks for offering to work on the HTML part, too - i think that'll be a GREAT new feature, so i'm really looking forward to that!

Feel free to open an "early"/WIP PR so we can collaborate if you need any help or got questions regarding the templating (not sure if you've used that before).

Thanks a lot for your feedback and support on all my PRs, I look forward to contributing more stuff in alignment with freqtrade. I'm learning as we move forward. I'll be looking more into templating with python, if you have any pointers or advice, it would be great if you can share them.

BTW, it was announced on Slack that the extended table was released to develop, but it's not accurate, only styling were released in this PR because I didn't commit the work I had on it for the extended tables.

Will open the new PR sometime this weekend, so we can start collaborating on it.

[hroff-1902]

BTW, it was announced on Slack that the extended table was released to develop, but it's not accurate, only styling were released in this PR because I didn't commit the work I had on it for the extended tables.

Oops, my bad, did not checked this in enough details, thought the pictures you posted were actual for this PR...