Return NaN for Indicators while within unstableBars?

[nimo23]

Besides Strategy#unstableBars, we now also have Indicator#getUnstableBars (according to #919).

As @team172011 suggested:

It is also necessary to determine the behavior of the Indicator.calculates(index) function regarding this issue. At the moment it is inconsistent. If it is not possible to calculate a value (e.g. because of index < size of timeFrame) some indicators return NaN, others zero while others modify timeFrame (make it smaller). I would suggest to return NaN whenever it is not possible to calculate the correct value

I think, this is a good idea. I would also suggest to return NaN whenever it is not possible to calculate the correct value of an indicator.

Any suggestions or objections on this topic?

Related issues:

• Introduce "unstable period" to indicators #101 (comment)
• Introduce "unstable period" to indicators #101 (comment)
• Thoughts and proposition for EMAIndicator #526 (comment)

[TheCookieLab]

Agreed, having this as consistent behavior would be ideal.

How do you see us implementing this - just returning NaN directly or should we use the UnstableIndicator to do so? I personally am partial to the simplicity of the former approach but am open to arguments for the latter.

Tangentially, we may also want to revisit how NaN behaves when compared to other Num values (ex see NumTest.testNaN). Right now it can be unstable when used in a sorting use case.

[nimo23]

How do you see us implementing this - just returning NaN directly or should we use the UnstableIndicator to do so?

We should just return NaN directly from the origin indicator. There is no need to wrap it by UnstableIndicator.

[team172011]

But when returning NaN the user does not have any change using or inspecting values for indices lower than the unstable bars value. Maybe there are cases where the user wants to use or at least the those values

[nimo23]

But when returning NaN the user does not have any change using or inspecting values for indices lower than the unstable bars value

Why should the user be allowed to use wrong indicator values? Why would the user even willingly want to use wrong indicator values? It can be dangerous to use wrong indicator values. It should be forbidden by default. I don't know any reasonable use case to use wrong indicator values.

But when returning NaN the user *does not have any change using or inspecting values for indices lower than the unstable bars value

What interest does the user have in observing incorrect indicator values ​​anyway?

In fact, I believe that many users do not even know that there are indicators that can initially give wrong indicator values ​​and users mistakenly classify them as correct. So it must be prevented to return wrong indicator values. I don't see any benefit in using or observing wrong indicator values. Or am I missing a benefit?

[hhashim1]

Users should never be presented with values during warm-up period as it does not provide any value to user and only creates confusion and bad decisions.

[hhashim1]

There should not be a wrapper around any indicator to for the purpose of excluding/suppressing data during warm-up period. This should be done by default by every indicator. Any major trading system and other technical analysis systems automatically suppress the warm up data.

[TheCookieLab]

@hhashim1 are you able/willing to take this on and create a PR?

[team172011]

@so following @nimo23 and @hhashim1 we should return NaN during warm up period?

This sounds easy and logical for Indicators like EMAIndicator with a barCount of X, returning NaN for all indices < X. But I also think about "unstable period" in this way: Why does my Indicator not match someone else’s values

Since EMA’s use the prior value in their calculation, the values depend on how many prior values you have. Related, there is some question as to how to initialize the first value of an EMA. The first problem is generally solved by including at least 200 Bars of TimeSeries data prior to the indices for which you are interested. The second problem is currently solved by setting the first EMA value to the first data value:

Indicators like StochasticRSIIndicator or even the EMAIndicator would need more bars than just the barCount. And this amount of data cannot be set to a fixed value, it depends on further factors than the barCount, such as other Indicators that are used by this Indicator and in the end on available bar history

@hhashim1 how does "any major trading system and other technical analysis systems" handle this? I bet there are different handlings of this issue

[nimo23]

And this amount of data cannot be set to a fixed value, it depends on further factors than the barCount, such as other Indicators that are used by this Indicator and in the end on available bar history

Well, we can have indictors like StochasticRSIIndicator or ADXIndicator (with Indicator#getUnstableBars > barCount). It propagates different unstableBars from nested indicator values simply by returning NaN,if at least one indicator returns NaN (e.g. Indicator1 returns a value != NaN but its nested Indicator returns NaN, hence at this index it returns NaN (e.g. 2 * NaN = NaN).

how does "any major trading system and other technical analysis systems" handle this? I bet there are different handlings of this issue

That's why it is difficult for one person to decide the unstableBars for ALL indicators. #944 was intended as an initial solution (in which a professional who knows the respective indicator better will improve the Indicator#unstableBars via a PR in the future). Most of the Indicator#unstableBars are correct, but I'm also not familiar with all indicators (and their warm-up phase) and therefore the Indicator#unstableBars value will very likely need to be improved in the future. For example: ADXIndicator, where the more bars are available, the more accurate the data becomes ("asymptotic approximation"). We should only go by the minimum principle here: either 0, barCount or an obvious fixed value for Indicator#unstableBars, which according to common practice is considered correct for the given indicator.

[hhashim1]

Take a look at the attached file which is the stochrsi indicator from talib. LOOKBACK_CALL used in lookbackSTOCHF = LOOKBACK_CALL(STOCHF)( optInFastK_Period, optInFastD_Period, optInFastD_MAType ); and other places is what does the warm-up period. This is the same process for all of the indicators. It depends on the indicator and back data is necessary and needs to be warmed up.
ta_STOCHRSI.txt

[team172011]

Maybe we have to distinguish?

• Unstable period -> fixed, defined by developer of indicator, does provide NaN values during this period?
• warm-up period -> variable, set by user (has a default?), indicator needs at least x consecutive previous bars to return valid data (return same as unstable period?)

Adding this to the system would produce a lot of checking for isUnstable and/or isNaN because ta4j allows to arbitrary combine indicators with other indicators. For example the EMAIndicator would have an unstable period and could be based on another StochasticRSIIndicator with an unstable period and a warm up period.
So, just returning NaN could lead to having NaN for all following indices as well because operations on NaN result always in NaN

open bugs that may be related to not having unstable/warum-up period
#971
#973

[nimo23]

warm-up period -> variable, set by user (has a default?), indicator needs at least x consecutive previous bars to return valid data (return same as unstable period?)

If the user wants to set a custom unstablePeriod, then the user can set Strategy#setUnstableBars. We could also have the method Indicator#setUnstableBars() as alternative..

So, just returning NaN could lead to having NaN for all following indices as well because operations on NaN result always in NaN

Correct, and I think, this is the correct way to handle it: It propagates different unstableBars from nested indicator values simply by returning NaN, if at least one indicator returns NaN (e.g. Indicator1 returns a value != NaN but its nested Indicator returns NaN, hence at this index it returns NaN (e.g. 2 * NaN = NaN).

Adding this to the system would produce a lot of checking for isUnstable and/or isNaN because ta4j allows to arbitrary combine indicators with other indicators.

Yes, but I think these checkings are necessary, because how else can we ensure that the indicator values ​​only return correct values?

[TheCookieLab]

Adding this to the system would produce a lot of checking for isUnstable and/or isNaN because ta4j allows to arbitrary combine indicators with other indicators. For example the EMAIndicator would have an unstable period and could be based on another StochasticRSIIndicator with an unstable period and a warm up period. So, just returning NaN could lead to having NaN for all following indices as well because operations on NaN result always in NaN

open bugs that may be related to not having unstable/warum-up period #971 #973

You're right simply propagating NaNs up is problematic because of this. Something to noodle on.

[nimo23]

FYI: https://www.reddit.com/r/algotrading/comments/bbnv8c/how_many_candles_minimum_to_get_an_accurate_ema/