Buy Signal not meet criteria after updating to 2021.1 release due to pandas upgrade

[jirbizz]

Hi,

After updating bot to 2021.1 version, a strategy that worked well before , right now is not working properly, throwing buy signals that doesn't meet criteria.

After install bot began to buy pairs and this was not the normal behaviour of this strategy.

Buy signal is using qtpylib and Bollinger bands and seems that is not calculating this indicator properly.

Has been an update of this library in last release?

Thanks for your support.

[xmatthias]

qtpylib was not touched in months, however other libraries are upgraded all the time.

I would not know of any that can cause a differently calculated indicator - nor have i ever heared about that before

[jirbizz]

Thanks for your response.

Its very strange, i'm sure it happened after update last version from 2020.12

I haven't modified strategy and I'm using BB indicator for buy signal. Freqtrade detects buy signal when criteria is not meet.

Well I'm going to investigate in deep.

Thanks

[Theagainmen]

Can you run a version of the bot from 2020.12 and 2021.1 with the exact same config and strategy file to verify the possible different behaviour (try backtesting and dry mode)?

[jirbizz]

Yes, I'm going to test this, but I can confirm that no changes have been made in config and strategy.

[xandross389]

Hi, a few days ago I seemed to have faced the same problem but I thought it was impossible and I did not give it much importance, I also use BB and RSI, however I can not assure it but I will be more alert in future buy signals and if there is a problem I will report it here.

I am using version 2020.12 and update all libraries to the latest version at the time of updating to version 2020.12

[AngryByte]

I also noticed the issue and I'm able to reproduce it easily.

1. Only add one pair with a low price (I used NPXS/ETH price around 0.00000052)
2. with the same config, strategy and market data, run backtesting with the same settings on the old and new versions.
3. Different results are observed.

in my case, the strategy uses Bollinger bands as

        bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
        dataframe['bb_lowerband'] = bollinger['lower']
        dataframe['bb_middleband'] = bollinger['mid']

I added a line to print the lower band column in populate_buy_trend and the numbers I saw were different, the old version gave correct data.

I can provide more logs as needed.

[xmatthias]

as said above, qtpylib was not changed, and had it's last change 5 months ago (you can check that yourself in https://github.com/freqtrade/freqtrade/tree/develop/freqtrade/vendor).

What has been changed are dependencies (specifically pandas / numpy), so you can try and downgrade these to the old versions one by one.
The more likely cause however is that a bug was fixed and that the old calculation was incorrect, with the new beeing correct (you think the old was correct because you based the strategy on top of that).

[AngryByte]

Hi,

I did some further investigations to confirm which version's data is to be considered correct, here is what I did:

1. Same data, config, and backtesting command for a single low price pair NPXS/ETH
2. modifed my strategy with the following lines to fully print the dataframe populate_buy_trend before it is returned

        compression_opts = dict(method='zip',archive_name = metadata['pair'].replace("/", "") + '-out.csv')  
        dataframe.to_csv('user_data/logs/' + metadata['pair'].replace("/", "")  + 'out.zip', index=False,
          compression=compression_opts,float_format='%.16f')

3. Ran the test using 2020.12 and 2021.1 and here is the dataframe that was saved:
   A. 2020.12:

date                      | open       | high       | low        | close      | volume | rsi          | bb_lowerband     | bb_middleband    | bb_upperband | buy | sell
2021-02-16 02:00:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 502601 | 58.314054475 | 5.183983494E-007 | 5.288333333E-007 | 5.392683172E-007 | 0 | 0
2021-02-16 02:05:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 41791 | 58.314054475 | 5.184484693E-007 | 5.296666667E-007 | 5.408848641E-007 | 0 | 0
2021-02-16 02:10:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 35108 | 58.314054475 | 5.187970489E-007 | 5.305E-007 | 5.422029511E-007 | 0 | 0
2021-02-16 02:15:00+00:00 | 0.00000053 | 0.00000053 | 0.00000053 | 0.00000053 | 369277 | 47.2780850763 | 5.1962035E-007 | 5.308333333E-007 | 0.000000542 | 0 | 0
2021-02-16 02:20:00+00:00 | 0.00000053 | 0.00000054 | 0.00000053 | 0.00000054 | 1142547 | 56.2040620902 | 0.00000052 | 5.313333333E-007 | 5.426553091E-007 | 0 | 0

And here's the new 2021.1

date                      | open       | high       | low        | close      | volume | rsi          | bb_lowerband     | bb_middleband    | bb_upperband | buy | sell
2021-02-16 02:00:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 502601 | 58.314054475 | 5.288333333E-007 | 5.288333333E-007 | 5.288333333E-007 | 0 | 0
2021-02-16 02:05:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 41791 | 58.314054475 | 5.296666667E-007 | 5.296666667E-007 | 5.296666667E-007 | 0 | 0
2021-02-16 02:10:00+00:00 | 0.00000054 | 0.00000054 | 0.00000054 | 0.00000054 | 35108 | 58.314054475 | 5.305E-007 | 5.305E-007 | 5.305E-007 | 0 | 0
2021-02-16 02:15:00+00:00 | 0.00000053 | 0.00000053 | 0.00000053 | 0.00000053 | 369277 | 47.2780850763 | 5.308333333E-007 | 5.308333333E-007 | 5.308333333E-007 | 0 | 0
2021-02-16 02:20:00+00:00 | 0.00000053 | 0.00000054 | 0.00000053 | 0.00000054 | 1142547 | 56.2040620902 | 5.313333333E-007 | 5.313333333E-007 | 5.313333333E-007 | 0 | 0
2021-02-16 02:25:00+00:00 | 0.00000054 | 0.00000054 | 0.00000053 | 0.00000054 | 865276 | 56.2040620902 | 5.318333333E-007 | 5.318333333E-007 | 5.318333333E-007 | 0 | 0

Notice how the Bollinger band upper, lower, and mid bands are all the same value which can't be right. I also confirmed the values from 2020.12 by comparing with tradingview

I understand that qtpylib was not updated, but I'm just trying to figure out why my strategy started misfiring after the new update.

I hope this helps, and keep up the good work.

[xmatthias]

Please do the test i explained above - take one version (say 2020.12) - install all dependencies as they are.

Ipdate freqtrade to 2021.1 - but do NOT upgrade dependencies (that's easy for a normally installed version - do a git pull - without running setup.sh or something similar).
With that, you'll be able to confirm if changes to freqtrade caused this.

If that's ok - then upgrade (one by one) the dependencies to the versions specified in requirements.txt (best start with pandas and numpy - as i suspect it's one of these as they are the most relevant for calculations in general).

This will help you identify which of the dependencies causes this.
After that (if you're still motivated by then) - do a step by step upgrade of the dependency from old to new - to identify which version in particular caused this (assuming there were multiple upgrades) - we can then (hopefully) identify the cause by reading the respective changelogs.

[jirbizz]

I'm pending to do the test with 2020.12 version. I will come back with results.

But I also have confirmed that BB values are not correct in 2021.1 version. Comparing trading view & buy positions , buy signal is ever thrown at middle band.

[AngryByte]

I did a slightly modified version of the test and narrowed down that pandas 1.2.2 is responsible for the issue with Bollinger bands.

here's what I did in details.

1. create a Dockerfile with version 2020.12 with a single update of pandas as follows:

FROM freqtradeorg/freqtrade:2020.12 
#freqtradeorg/freqtrade:master

RUN pip install pandas==1.2.2

ENTRYPOINT ["freqtrade"]
# Default to trade mode
CMD [ "trade" ]

2. I built the image and used it for backtesting.

debian@machine:/opt/apps/testversion$ sudo docker build -t testfreq .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM freqtradeorg/freqtrade:2020.12
2020.12: Pulling from freqtradeorg/freqtrade
6ec7b7d162b2: Already exists
80ff6536d04b: Pull complete
....
Digest: sha256:1038d61f6ee4fb071e90c1c54b2b539d1dca9c5e11a9715cf39f7ea14c22d87a
Status: Downloaded newer image for freqtradeorg/freqtrade:2020.12
 ---> 388c60ac91cb
Step 2/4 : RUN pip install pandas==1.2.2
 ---> Running in 410290c29ec0
Collecting pandas==1.2.2
  Downloading pandas-1.2.2-cp38-cp38-manylinux1_x86_64.whl (9.7 MB)
Requirement already satisfied: numpy>=1.16.5 in /root/.local/lib/python3.8/site-packages (from pandas==1.2.2) (1.19.4)
Requirement already satisfied: python-dateutil>=2.7.3 in /root/.local/lib/python3.8/site-packages (from pandas==1.2.2) (2.8.1)
Requirement already satisfied: pytz>=2017.3 in /root/.local/lib/python3.8/site-packages (from pandas==1.2.2) (2020.5)
Requirement already satisfied: six>=1.5 in /root/.local/lib/python3.8/site-packages (from python-dateutil>=2.7.3->pandas==1.2.2) (1.15.0)
Installing collected packages: pandas
  Attempting uninstall: pandas
    Found existing installation: pandas 1.1.5
    Uninstalling pandas-1.1.5:
      Successfully uninstalled pandas-1.1.5
Successfully installed pandas-1.2.2

3. The results where wrong for Bollinger band upper and lower bands equivalent to those of 2021.1
4. I did an additional test only updating numpy to 1.20.1 and the results were correct and equivalent to stable 2020.12

[xmatthias]

the big question is - which of the versions of pandas in between this (there should be 1.2.0 and 1.2.1` ) are the first to cause this.

The big suspect is obviously be 1.2.0 - beeing a bigger release than the 2 bugfix releases, and the changelog has some relevant points:

Rolling.mean() and Rolling.sum() use Kahan summation to calculate the mean to avoid numerical problems (GH10319, GH11645, GH13254, GH32761, GH36031)

and

Rolling.var() and Rolling.std() use Kahan summation and Welford’s Method to avoid numerical issues (GH37051)

source

To further investigate this, i think you'll have to "untie" this problem from freqtrade (you've just shown that it's not freqtrade that's causing it anyway) - and investigate this with "pandas only" methods to be able to bug-report against the pandas repository.

what does a manually calculated bollinger band tell you when doing this same comparison?

should be as simple as

dataframe['middleband'] = dataframe['close'].rolling(window).mean()
rolling_std = dataframe['close'].rolling(window).std()
dataframe['upperband'] = dataframe['middleband'] + (2 * rolling_std)
dataframe['lowerband'] = dataframe['middleband'] - (2 * rolling_std)

While this won't return exactly the same as the default qtpylib version (which uses typicalprice in most examples) - it should still give different results between the 2 versions.

[xmatthias]

Based on the above, it's related to the pandas update to pandas 1.2 (which does list some changes to how mean() and std is calculated).

Apparently it's kindof an expected issue with pandas (pandas-dev/pandas#39872, pandas-dev/pandas#37051) - but without providing any workaround or solution other than downgrading.

[adverserath]

but do you lose any fix between upgrading? Could we not multiple all of the existing data by 1000 then divide the output once the calculation is done?

[xmatthias]

You can do that for sure, but it's a fix everyone needs to apply within their strategy, not something that'll be done automatically.

[AngryByte]

Following up on the First version that introduced the BB issue, I tested pandas 1.1.5 (used in freqtrade 2020.12), 1.2.0rc0, and up to 1.2.2

All versions after 1.1.5 had wrong values for the upper/lower BBs for pairs with low prices.

I will try to put together a case with Pandas, but I don't see this getting fixed soon. I'm wondering if we can consider rolling back to Pandas 1.1.5 (not sure about other dependencies) for the benefit of the freqtrade community until the issue with pandas is resolved.

[xmatthias]

well the "case" for pandas is in the 2 issues i linked above - but reading through these it doesn't seem like they consider this as a problem, but as a feature ...

downgrading is a possibility in theory - but assuming they'll never fix this as they don't consider it a bug - at what point can we upgrade again to get new features / speed improvements / whatever ... ?

in general, pandas will work fine with non-BTC pairs - which (from reading on discord) seems to be more common anyway...

[robcaulk]

I am not observing this behavior with dev branch + all pandas versions and using low value BTC pairs.

[edit] I wasn't looking at the full precision. After displaying the full float, it appears the bands are being calculated correctly.

[xmatthias]

@robcaulk how are you calculating bollinger bands? it's a quite simple, standard indicator, which is available in basically every library (qtpylib, ta-lib, ...) - not all use pandas for these calculations.

[robcaulk]

@robcaulk how are you calculating bollinger bands? it's a quite simple, standard indicator, which is available in basically every library (qtpylib, ta-lib, ...) - not all use pandas for these calculations.

I tested both qtpylib and ta-lib with pandas 1.1.5 and pandas 1.2.2. I am running Ubuntu 18.04. The "low value pairs" are roughly 1e-5 btc.

In all cases I did not observe the described problem :-/

[skhalsa10]

but do you lose any fix between upgrading? Could we not multiple all of the existing data by 1000 then divide the output once the calculation is done?

Hey dude this is brilliant. I am somewhat new to pandas and so I didn't really know how to accomplish what you suggested. but I finally figured it out and I actually tend to like this. it allows you to use to latest and greatest and still salvage a strategy that utilizes Bollinger Bands and probably any other strategy that may rel averages and std using btc and cheap coins. but in case the next person is like ME who was reading this and had no idea how to implement it here is how I got it to work. Obviously some precision is lost but I think its good enough:

        dataframe['open'] = dataframe['open'].apply(lambda x: x*1000)
        dataframe['close'] = dataframe['close'].apply(lambda x: x*1000)
        dataframe['high'] = dataframe['high'].apply(lambda x: x*1000)
        dataframe['low'] = dataframe['low'].apply(lambda x: x*1000)

        bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(dataframe), window=20, stds=2)
        dataframe['bb_lowerband2'] = bollinger['lower']/1000
        dataframe['bb_middleband2'] = bollinger['mid']/1000
        dataframe['bb_upperband2'] = bollinger['upper']/1000

        dataframe['open'] = dataframe['open'].apply(lambda x: x/1000)
        dataframe['close'] = dataframe['close'].apply(lambda x: x/1000)
        dataframe['high'] = dataframe['high'].apply(lambda x: x/1000)
        dataframe['low'] = dataframe['low'].apply(lambda x: x/1000)

[xmatthias]

considering that the precision is around e-16, you'll not lose much as the lowest price i've seen on an exchange is e-8 (they usually don't support any lower precision than e-6 ...) - so you should be fine.

The probably better method for bollinger bands might be to use ta.BBANDS ... which should not suffer from this problem as it's not using pandas for the calculation at all.

[skhalsa10]

@xmatthias thanks for this alternative as well!

[samgermain]

This is how to use ta.BBANDS for anyone else wondering

        boll = ta.BBANDS(dataframe, nbdevup=2.0, nbdevdn=2.0, timeperiod=15)  #set timeperiod to your time period
        dataframe['bb_lower'] = boll['lowerband']
        dataframe['bb_middle'] = boll['middleband']
        dataframe['bb_upper'] = boll['upperband']

[skhalsa10]

This is how to use ta.BBANDS for anyone else wondering

        boll = ta.BBANDS(dataframe, nbdevup=2.0, nbdevdn=2.0)
        dataframe['bb_lower'] = boll['lowerband']
        dataframe['bb_middle'] = boll['middleband']
        dataframe['bb_upper'] = boll['upperband']

Dude thanks this is helpful!! What is the window that it calculates from? 20?

[samgermain]

@skhalsa10 I don't actually know what windows are, but there's one more argument called matype which I don't understand, but it's default value is 0

[xmatthias]

it's quite unfortunate that the ta-lib documentation is quite lacking in many regards.

matype will (most likely) be either using SMA or EMA (or something else) ... but without documentation what each type(numbe) means - it'll be rather difficult to deviate from the default.

[wr0ngc0degen]

@samgermain weird, I tried using ta.BBANDS and it doesn't work for me as expected. for small values it gives the same number (I'm keeping the "incorrect" pandas version - in particular 1.2.3 and my TA-lib version is 0.4.19). and test_talib_bollingerbands_near_zero_values is failing for me. what I might be missing?
although the fix suggested by @skhalsa10 seems to work fine.
@xmatthias, do you think it makes sense to add the latter to the template files (and well, basically all the places where qtpylib.bollinger_bands is used?

[xmatthias]

if the ta-lib test fails, then you installed ta-lib wrongly.

We need to compile the C dependency with a small change to have it work for the very low numbers BTC pairs generate.

[wr0ngc0degen]

@xmatthias hmmmm, you mean this one, right?
sed -i.bak "s|0.00000001|0.000000000000000001 |g" src/ta_func/ta_utility.h
I'm sure I was following these steps when installing. I've even double-checked my terminal history and it's there...
and I was too fast to say that the other change works fine. The BB calculation works fine but then I'm getting the "Dataframe returned from strategy has mismatching last close price" warning that supposedly has to do with precision and is likely to be an easy adjustment

[xmatthias]

Dataframe returned from strategy has mismatching last close price

this simply means that you assigned something to "close" - which you really shouldn't.
it's got nothing to do with precisions as such.

[wr0ngc0degen]

@xmatthias I just uninstalled ta-lib and re-installed it using install_ta-lib.sh and still no luck, the test is failing. any suggestion?

btw it looks to me that it needs to be executed from the build_helpers folder otherwise it logically fails with Error opening archive: Failed to open 'ta-lib-0.4.0-src.tar.gz' cause we're still in the upper directory

this simply means that you assigned something to "close" - which you really shouldn't.
it's got nothing to do with precisions as such.

got it

[xmatthias]

uninstalled ta-lib

make sure to uninstall the C level dependency as well, not just the python library - otherwise the installation will not really do much ...

[wr0ngc0degen]

sure. what I did is

wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz
tar xvzf ta-lib-0.4.0-src.tar.gz
cd ta-lib
./configure --prefix=/usr/local
make
sudo make uninstall

and then run ./install_ta-lib.sh from build_helpers folder

[xmatthias]

well you'll need to remove both the python dep - as well as the c library.

When i was in a similar hole - i removed it all (first C, then pip uninstall ta-lib) - followed by ./install_ta-lib.sh - and then pip install ta-lib ...

[wr0ngc0degen]

tried this several times no luck. oh boy that's quite annoying. and I'm pretty sure that I've done ta-lib installation as instructed from the first time. oh well.
from the top of you head, does it affect any (widely used) indicator calculation other than Bollinger bands calculations for BTC pairs (ie small numbers)?
my guess is that it only concerns indicators that do calculate "squared prices" and nothing obvious comes to mind but I might be missing something.

so far the workaround with BB for me is the following:

        bb_input = dataframe[['high', 'low', 'close']].copy()
        bb_input = bb_input[['high', 'low', 'close']].apply(lambda x: x*1000)
        bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(bb_input), window=20, stds=2)
        dataframe['bb_lowerband'] = bollinger['lower'] / 1000
        dataframe['bb_middleband'] = bollinger['mid'] / 1000
        dataframe['bb_upperband'] = bollinger['upper'] / 1000`

[lucesgabriel]

Hi wr0ngc0degen / Team,
I faced the same issue. I tried many things to fix the problem but I could not.

the workaround with BB was the same wr0ngc0degen did.

    bb_input = dataframe[['high', 'low', 'close']].copy()
    bb_input = bb_input[['high', 'low', 'close']].apply(lambda x: x*1000)
    bollinger = qtpylib.bollinger_bands(qtpylib.typical_price(bb_input), window=20, stds=2)
    dataframe['bb_lowerband'] = bollinger['lower'] / 1000
    dataframe['bb_middleband'] = bollinger['mid'] / 1000
    dataframe['bb_upperband'] = bollinger['upper'] / 1000

[xmatthias]

Closing this as the update to pandas 1.3.0 (#5240) finally fixes this upstream bug release-log link.