Back test results are different between cached data and live data

[none2003]

Hi edtechre:

I'm just playing the notebook: https://github.com/edtechre/pybroker/blob/master/docs/notebooks/10.%20Rotational%20Trading.ipynb

I can reproduce the same result when modifying nothing to the code, however, if I add following line to the first cell of notebook, the result is different from first run nor the result in online doc.

pyb.enable_caches('rotational_trading')

the first 10 row result using cached data:
1,buy,META,2018-01-03,275,NaN,183.06,0.0
2,buy,NFLX,2018-01-03,243,NaN,203.86,0.0
3,sell,META,2018-02-01,275,NaN,191.61,0.0
4,buy,AMD,2018-02-01,3903,NaN,13.53,0.0
5,sell,AMD,2018-02-05,3903,NaN,11.56,0.0
6,buy,AMZN,2018-02-05,649,NaN,69.49,0.0
7,sell,NFLX,2018-04-03,243,NaN,284.63,0.0
8,sell,AMZN,2018-04-03,649,NaN,69.23,0.0
9,buy,INTC,2018-04-03,1151,NaN,49.18,0.0
10,buy,MSFT,2018-04-03,636,NaN,88.97,0.0

the first 10 row result using live data:
1,buy,AAPL,2018-01-03,1161,NaN,43.31,0.0
2,buy,AMD,2018-01-03,4231,NaN,11.75,0.0
3,sell,AAPL,2018-02-01,1161,NaN,41.92,0.0
4,buy,NFLX,2018-02-01,181,NaN,267.67,0.0
5,sell,AMD,2018-02-05,4231,NaN,11.56,0.0
6,buy,AMZN,2018-02-05,707,NaN,69.49,0.0
7,sell,AMZN,2018-04-03,707,NaN,69.23,0.0
8,sell,NFLX,2018-04-03,181,NaN,284.63,0.0
9,buy,INTC,2018-04-03,1014,NaN,49.18,0.0
10,buy,MSFT,2018-04-03,560,NaN,88.97,0.0

Is this normal?

[edtechre]

Hi @none2003,

Thanks for pointing this out! The notebook should include an additional check that 20 bars have been completed:

def rank(ctxs: dict[str, ExecContext]):
    if tuple(ctxs.values())[0].bars <= 20:
        return
    scores = {
        symbol: ctx.indicator('roc_20')[-1]
        for symbol, ctx in ctxs.items()
    }
    sorted_scores = sorted(
        scores.items(), 
        key=lambda score: score[1],
        reverse=True
    )
    threshold = pyb.param('rank_threshold')
    top_scores = sorted_scores[:threshold]
    top_symbols = [score[0] for score in top_scores]
    pyb.param('top_symbols', top_symbols)

def rotate(ctx: ExecContext):
    if ctx.bars <= 20:
        return
    if ctx.long_pos():
        if ctx.symbol not in pyb.param('top_symbols'):
            ctx.sell_all_shares()
    else:
        target_size = pyb.param('target_size')
        ctx.buy_shares = ctx.calc_target_shares(target_size)
        ctx.score = ctx.indicator('roc_20')[-1]

What's happening is that caching does not maintain the order of the rows of the original DataFrame. During the initial 20 bars of the backtest, the 20 day ROC score will be NaN, so the order of the scores will depend on the order that the symbols appear in the DataFrame. That ordering becomes non-deterministic once the cached data is read.

The issue is with the notebook code, not the caching mechanism. I have updated the notebook with the fix:
https://www.pybroker.com/en/latest/notebooks/10.%20Rotational%20Trading.html

Let me know if you need anything else.

[none2003]

Hi @none2003,

Thanks for pointing this out! The notebook should include an additional check that 20 bars have been completed:

def rank(ctxs: dict[str, ExecContext]):
    if tuple(ctxs.values())[0].bars <= 20:
        return
    scores = {
        symbol: ctx.indicator('roc_20')[-1]
        for symbol, ctx in ctxs.items()
    }
    sorted_scores = sorted(
        scores.items(), 
        key=lambda score: score[1],
        reverse=True
    )
    threshold = pyb.param('rank_threshold')
    top_scores = sorted_scores[:threshold]
    top_symbols = [score[0] for score in top_scores]
    pyb.param('top_symbols', top_symbols)

def rotate(ctx: ExecContext):
    if ctx.bars <= 20:
        return
    if ctx.long_pos():
        if ctx.symbol not in pyb.param('top_symbols'):
            ctx.sell_all_shares()
    else:
        target_size = pyb.param('target_size')
        ctx.buy_shares = ctx.calc_target_shares(target_size)
        ctx.score = ctx.indicator('roc_20')[-1]

What's happening is that caching does not maintain the order of the rows of the original DataFrame. During the initial 20 bars of the backtest, the 20 day ROC score will be NaN, so the order of the scores will depend on the order that the symbols appear in the DataFrame. That ordering becomes non-deterministic once the cached data is read.

The issue is with the notebook code, not the caching mechanism. I have updated the notebook with the fix: https://www.pybroker.com/en/latest/notebooks/10.%20Rotational%20Trading.html

Let me know if you need anything else.

Thank you for your quick action!

I'm a little confuse about "caching does not maintain the order of the rows of the original DataFrame", so how pybroker make sure the time sequence when get data from cache? Is data get resorted when retrieve from cache when using?

[edtechre]

PyBroker does maintain the ordering of the dates, so the time sequence will be correct. However, the order in which the rows appear for different symbols on the same date does not have a guaranteed ordering.