-
Notifications
You must be signed in to change notification settings - Fork 239
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Back test results are different between cached data and live data #20
Comments
Hi @none2003, Thanks for pointing this out! The notebook should include an additional check that 20 bars have been completed:
What's happening is that caching does not maintain the order of the rows of the original DataFrame. During the initial 20 bars of the backtest, the 20 day ROC score will be NaN, so the order of the scores will depend on the order that the symbols appear in the DataFrame. That ordering becomes non-deterministic once the cached data is read. The issue is with the notebook code, not the caching mechanism. I have updated the notebook with the fix: Let me know if you need anything else. |
Thank you for your quick action! I'm a little confuse about "caching does not maintain the order of the rows of the original DataFrame", so how pybroker make sure the time sequence when get data from cache? Is data get resorted when retrieve from cache when using? |
PyBroker does maintain the ordering of the dates, so the time sequence will be correct. However, the order in which the rows appear for different symbols on the same date does not have a guaranteed ordering. |
Hi edtechre:
I'm just playing the notebook: https://github.com/edtechre/pybroker/blob/master/docs/notebooks/10.%20Rotational%20Trading.ipynb
I can reproduce the same result when modifying nothing to the code, however, if I add following line to the first cell of notebook, the result is different from first run nor the result in online doc.
pyb.enable_caches('rotational_trading')
the first 10 row result using cached data:
1,buy,META,2018-01-03,275,NaN,183.06,0.0
2,buy,NFLX,2018-01-03,243,NaN,203.86,0.0
3,sell,META,2018-02-01,275,NaN,191.61,0.0
4,buy,AMD,2018-02-01,3903,NaN,13.53,0.0
5,sell,AMD,2018-02-05,3903,NaN,11.56,0.0
6,buy,AMZN,2018-02-05,649,NaN,69.49,0.0
7,sell,NFLX,2018-04-03,243,NaN,284.63,0.0
8,sell,AMZN,2018-04-03,649,NaN,69.23,0.0
9,buy,INTC,2018-04-03,1151,NaN,49.18,0.0
10,buy,MSFT,2018-04-03,636,NaN,88.97,0.0
the first 10 row result using live data:
1,buy,AAPL,2018-01-03,1161,NaN,43.31,0.0
2,buy,AMD,2018-01-03,4231,NaN,11.75,0.0
3,sell,AAPL,2018-02-01,1161,NaN,41.92,0.0
4,buy,NFLX,2018-02-01,181,NaN,267.67,0.0
5,sell,AMD,2018-02-05,4231,NaN,11.56,0.0
6,buy,AMZN,2018-02-05,707,NaN,69.49,0.0
7,sell,AMZN,2018-04-03,707,NaN,69.23,0.0
8,sell,NFLX,2018-04-03,181,NaN,284.63,0.0
9,buy,INTC,2018-04-03,1014,NaN,49.18,0.0
10,buy,MSFT,2018-04-03,560,NaN,88.97,0.0
Is this normal?
The text was updated successfully, but these errors were encountered: