Resampling not working correctly

[ghost]

Hi there,
I have noticed the following:
Example Case Resampling 10m period (resampling from 5m) on Bitmex (implemented analog to the example for 15m)

Each time a new candle is created (every 10m), the new candle in the database table "candlesticks" will have the close value of the previous(!) 5m candle and not the actual close of the 10m period (or the second 5m candle).
This then causes follow up problems with incorrect indicators etc.
It looks to me like that the program does not have the latest 5m candles available during runtime when the resampling methods is called.

When I restart the program, it updates the candlesticks 10m period table with the lates and correct value.
Any idea what could cause this issue?

Edit: I am assuming this is related to for example Bitmex.js line around 230ff

                    setTimeout(async () => {
                        if (resamples[symbol['symbol']] && resamples[symbol['symbol']][period] && resamples[symbol['symbol']][period].length > 0) {
                            for (let periodTo of resamples[symbol['symbol']][period]) {
                                await me.candlestickResample.resample(this.getName(), symbol['symbol'], period, periodTo, true)
                            }
                        }
                    }, 1000);```

[ghost]

I have to correct myself. I made a change to the code in another place to load up to 50000 candles from the database. This caused the later update etc. to take longer, resulting in the resampling not finding an up to date database! I reset the values to the original and it now works from what I can see.

[ghost]

Well, sorry for the back and forth. But actually it does still not work! Regularly the resampled 10m candle is just using the close price of the first 5m candle but ignores the second candle, therefore causing wrong results!

Additional Issue: When the new resampled candle is finished the ticker event is triggered before the candle is actually in the database it seems. That causes the bot to have the wrong data for the indicators (unfinished resample candle) during the first ticker event that comes up after the new candle got created.
On the second ticker event after that it is correct again then I think.
Maybe the resampling should not be async?

[ghost]

I was able to solve the first issue (10m candles with outdated data from the previous 5m candle) by increasing the timeout in bitmex.js around line 230 to 5000 (from 1000).

                    // lets wait for settle down of database insert: per design we dont know when is was inserted to database
                    setTimeout(async () => {
                        if (resamples[symbol['symbol']] && resamples[symbol['symbol']][period] && resamples[symbol['symbol']][period].length > 0) {
                            for (let periodTo of resamples[symbol['symbol']][period]) {
                                await me.candlestickResample.resample(this.getName(), symbol['symbol'], period, periodTo, true)
                            }
                        }
                    }, 5000);

The second issue solution: As the database takes some time to update the resampled candle with the final data, there will be one ticker event/one execution after the 5m candle update where the system would use the then still unfinished candle for indicator calculations. I have set the strategy up to only use candles that are at least 35 seconds old. That way I ignore the unfinished candles (but don't have to wait 5m for the next update).

In general I would hope there should be a better solution to not use these long timeouts of 5 or 35 seconds! Especially when thinking about faster candles I would think this could maybe have a better solutions.
Any ideas about this?

Thank you!

[Haehnchen]

sry, yes indeed it was just a quick hack; refactored it now to use Promise under the hood.

can you maybe also check also timing issues. i think is was fixed with this commit some days ago: 42bce1c

This resolved gabs issues on backtesting only Bitmex (i think this what you noticed)

[ghost]

Thank you for your reply!
Unfortunately even with the newest commits it is not working. After a candle change it will still use the value from the previous 5m candle (technically the unfinished 10m candle). Instead it should only trigger the tick event when the 10m candle is completed I would say.
I think this happens because as soon as the timestamp from the unfinished/future 10m candle is in the present, it will take the value for the next tick, even though the candle is still not finished/waiting for the final update of data.

I like your change with the async handling a lot in the Bitmex.js. Looks much better than with the timeout! :) However, if I am not mistaken you only changed the initial loading of candles from the exchange to async. The regular "stream" of candles is still using the timeout (around line 250 in Bitmex.js). If that would be async as well, it would be lovely I think! :)

[ghost]

Awesome! 👏