[R-Forge #2043] DPO implementation looks into the future

[joshuaulrich]

Submitted by: Mirko Fit
Assigned to: Joshua Ulrich
R-Forge link

The implementation of the detrended price oscillator looks into the future by shifting moving averages backwards.

Why this is problematic:

1.) The current values of DPO contain information about the future
-> I used genetic programming to find trading rules in forex markets and
found a lot of unrealisticly successful trading rules with that approach.
By now it looks like the future information within DPO made it easy
to find trading rules with high forecasting abilities.

2.) It is impossible to use it in an event based system, as the information of future MA values is not available at calculation time

From my point of view, every indicator that produces n/a values at the end of the time series, may contain future information and is therefore not practically usable.

In case of the DPO, a hotfix could be to shift the final result forward by 'shift' (option).
The most clean I can imagine is to implement it via some buffer (like shift registers in hardware). One would create a buffer of size 'shift' and store the past prices in there. When the new value of DPO is calculated, one can substract the MA value from the price of 'shift' periods ago.

Followups:

Date: 2013-02-24 23:59
Sender: Joshua Ulrich
This isn't a bug; it's how the indicator is calculated. See the references below:
http://www.fmlabs.com/reference/DPO.htm
http://www.equis.com/Customer/Resources/TAAZ/?c=3&p=48
http://stockcharts.com/school/doku.php?id=chart_school:technical_indicators:detrended_price_osci

[systats]

Hi Joshua,

i wonder which value it has to incorporate it into a model as it is leaking the future? Its seems also to be the only obviously leaking feature. Maybe renaming it as looking_future_DPO prevents use in the first place. Everything else works fine for me so far. Thank you!

[joshuaulrich]

Hi @systats, I don't think I should have included DPO() in TTR for that reason, but I think it's too late to remove it or change the name now.

Maybe I could add a message about it looking into the future the first time it's called... with a global option to turn it off by default... but that could also be annoying too. I might be able to find a way to have the message thrown once per TTR install, with the same global option to disable it completely.

Thoughts?

[braverock]

It seems to me that the warning in the documentation about look-ahead-bias is sufficient.

Plenty of published papers make the mistake of demeaning the entire series, for example. It makes the inference suspect, certainly, but it is just a 'detrending' methodology.

Likewise, the DPO implementation's use of a smoother (a two-sided filter) rather than a one-sided backwards looking filter is just the way it is specified.

Technical analysis models are more useful as starting points for developing something modern anyway than they are as models on their own...