backtesting issue in returning a resample dataframe

[maxma-bit]

The strategy ReinforcedAverageStrategy in the folder Berlinguyinca https://github.com/freqtrade/freqtrade-strategies/tree/master/user_data/strategies/berlinguyinca works perfectly in real-mode and dry-mode but does not work with backtesting. This stradegy manipulates the dataframe before returning it to the populate_buy_trend method, namelly adding the column called resample_sma

I noticed that if you print the dataframe in backtesting mode, the column called resample_sma follows the buy columns in the resulting dataframe, see:

['date', 'open', 'high', 'low', 'close', 'volume', 'maShort', 'maMedium', 'bb_lowerband', 'bb_upperband', 'bb_middleband', 'buy', 'sell', 'resample_sma']
date open high ... buy sell resample_sma
475 2020-01-09 22:45:00+00:00 0.000028 0.000029 ... 0 0 NaN
476 2020-01-09 23:00:00+00:00 0.000029 0.000029 ... 0 0 NaN
477 2020-01-09 23:15:00+00:00 0.000028 0.000028 ... 0 0 NaN
478 2020-01-09 23:30:00+00:00 0.000028 0.000028 ... 0 0 NaN
479 2020-01-09 23:45:00+00:00 0.000028 0.000028 ... 0 0 NaN

[5 rows x 14 columns]
['date', 'open', 'high', 'low', 'close', 'volume', 'maShort', 'maMedium', 'bb_lowerband', 'bb_upperband', 'bb_middleband', 'buy', 'sell', 'resample_sma']
date open high ... buy sell resample_sma
475 2020-01-09 22:45:00+00:00 0.001830 0.001835 ... 0 0 NaN
476 2020-01-09 23:00:00+00:00 0.001834 0.001835 ... 0 0 NaN
477 2020-01-09 23:15:00+00:00 0.001833 0.001836 ... 0 0 NaN
478 2020-01-09 23:30:00+00:00 0.001836 0.001839 ... 0 0 NaN
479 2020-01-09 23:45:00+00:00 0.001838 0.001844 ... 0 0 NaN

while the same strategy ReinforcedAverageStrategy in dry-mode returns a correct dataframe with the columns buy and sell at the end. See below:

[5 rows x 14 columns]
['date', 'open', 'high', 'low', 'close', 'volume', 'maShort', 'maMedium', 'bb_lowerband', 'bb_upperband', 'bb_middleband', 'resample_sma', 'buy', 'sell']
date open high ... resample_sma buy sell
494 2020-01-15 20:30:00+00:00 0.000277 0.000278 ... NaN NaN NaN
495 2020-01-15 20:45:00+00:00 0.000276 0.000276 ... NaN NaN NaN
496 2020-01-15 21:00:00+00:00 0.000276 0.000278 ... NaN NaN NaN
497 2020-01-15 21:15:00+00:00 0.000277 0.000277 ... NaN NaN NaN
498 2020-01-15 21:30:00+00:00 0.000275 0.000276 ... NaN NaN NaN

[5 rows x 14 columns]
['date', 'open', 'high', 'low', 'close', 'volume', 'maShort', 'maMedium', 'bb_lowerband', 'bb_upperband', 'bb_middleband', 'resample_sma', 'buy', 'sell']
date open high ... resample_sma buy sell
494 2020-01-15 20:30:00+00:00 0.000002 0.000002 ... NaN NaN NaN
495 2020-01-15 20:45:00+00:00 0.000002 0.000002 ... NaN NaN NaN
496 2020-01-15 21:00:00+00:00 0.000002 0.000002 ... NaN NaN NaN
497 2020-01-15 21:15:00+00:00 0.000002 0.000002 ... NaN NaN 1.0
498 2020-01-15 21:30:00+00:00 0.000002 0.000002 ... NaN NaN NaN

I tried different variation of returning a resampled dataframe to populate_buy_trend method, but I have the same problem described above in the case of backtesting, any suggestions?

[xmatthias]

I'll have a look at it tonight.

Any particular pair you're looking at - or is it "pair-agnostic" ... ?

[maxma-bit]

it seems pair-agnostic. I tried with a dozen of different pairs.

[xmatthias]

Now i've moved this issue to the strategy repository, as i think it's better suited here, as the problem is with the strategy - not with freqtrade itself.

I've also opened a PR with the (#63) fixes applied.

I think it's accidental that it never worked in backtesting but did work during dry-run. Since the dataframe did not fill na values after merging, it was causing only accidental buys at the resampled timeframe.
However i think realistically, even when using longer / resampled timeframes, the cross may also apply between the 2d mark - so filling NA values is realistic.

Also, by using interpolate(method='time'), the strategy was looking into the future - since this is supposed to smoothen the courve (but we don't know the next value yet - so it's telling us if it's moving up or down before the next candle closes.

[hroff-1902]

@maxma-bit the fixes were merged,
pls update both technical and freqtrade-strategies repos and test the solution.

[maxma-bit]

I still can't get the strategy ReinforcedAverageStrategy to work. I updated technical, the strategies and also the container. And backtested again. I'm getting the KeyError: 'resample_2880_sma', see:

2020-02-07 09:47:10,182 - freqtrade.optimize.backtesting - DEBUG - Run backtest, stake_amount: 14, start_date: 2020-01-15T00:00:00+00:00, end_date: 2020-01-17T23:45:00+00:00, max_open_trades: 2, position_stacking: False
2020-02-07 09:47:10,184 - freqtrade.strategy.interface - DEBUG - Populating buy signals for pair ADA/USDT.
2020-02-07 09:47:10,185 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1622, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'resample_2880_sma'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 44, in start_backtesting
    backtesting.start()
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 408, in start
    all_results[self.strategy.get_strategy_name()] = self.backtest(
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 310, in backtest
    ticker: Dict = self._get_ticker_list(processed)
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 166, in _get_ticker_list
    self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
  File "/freqtrade/freqtrade/strategy/interface.py", line 480, in advise_buy
    return self.populate_buy_trend(dataframe, metadata)
  File "/freqtrade/user_data/backtest_results/2020-02-07__09:47:06/_batman/USDT_short_15m/strategies/ReinforcedAverageStrategy.py", line 76, in populate_buy_trend
    (dataframe['close'] > dataframe['resample_2880_sma']) &
  File "/usr/local/lib/python3.8/site-packages/pandas/core/frame.py", line 2800, in __getitem__
    indexer = self.columns.get_loc(key)
  File "/usr/local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 2648, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 111, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1622, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'resample_2880_sma'
2020-02-07 09:47:10,245 - freqtrade.exchange.exchange - DEBUG - Exchange object destroyed, closing async loop
root@446a50fed845:/freqtrade/scripts#

[hroff-1902]

(^^ formatted)

[xmatthias]

Well this strategy uses 4h candles by default - and the resampling used is timeframe_to_minutes(self.ticker_interval) * 12.

If you're using different candles (1h / lower) you need to adjust the key.

the calculation is rather simple - it's 4h (4 * 60 * 12 == 2880).

Now obviously, if you're using 1h candles, you'll need 60 * 12 = 720 - your key is therefore resample_720_sma.

Hope this makes sense? unfortunately, the logs don't show which interval you're actually using.