Adjusting queue position model and latencies

[artiko88]

Hi!

I am noticing about 20% more trades being simulated in backtester when compared to live trading.
I am fairly confident that i am able to achieve the same values for bid and ask orders between live and backtest and at least judging visually on lineplots they match nicely. Market features that are used to obtain those bid/ask prices do match as well.

However, getting so much less order fills during live trading concerns me. Perhaps, i should tinker with queue position models and latencies. While with latencies i can play with multipliers and interpolated latencies, how does one adjusts queue models aside from trying several of them? There are models that accept n as an argument, like PowerProbQueueModel(n) and its variation - what direction should n be moved towards in order to get pessimistic probability of getting a fill?

[nkaz001]

For a pessimistic fill, I would suggest using PowerProbQueueModel3 with a higher n value.

Please see https://hftbacktest.readthedocs.io/en/latest/order_fill.html#probqueuemodel

Which exchange are you backtesting for? With Binance Futures, bookTicker stream (TOB) provides more frequent updates. You might need data fusion of L2 and TOB to get the finest and frequent data for more precise backtesting.

[artiko88]

Will check PowerProbQueueModel3 with different n, thank you for suggesting! I am backtesting for binance futures and use tardis data preprocessed with the tool provided with hftbacktest. Are you implying that on top of trades and book updates one might require TOB events as well? This sounds confusing. Aren't we capturing all bbo events with book updates which hftbacktest performs internally? In such case, this may explain some missing data that i am experiencing in other back tests. How does one fuse this feed with usual trades/book_updates? Edit: Found 'quotes' datatype on Tardis which is TOB events. But according to the description of L2 datafeed, TOB events are already included in it. If one decides to fuse quotes with L2/trades by modifying tardis converter, should we handle TOB events like DEPTH event in the converter code?

[nkaz001]

If you examine the raw data from Binance Futures, you'll notice that the depth stream is indeed conflated even in the '@0ms' stream based on update ID changes. Additionally, comparing the BBO from the bookTicker stream with that from the depth stream can confirm this. While I'm unsure how Tardis manages the data, I highly recommend investigating the raw data for further insights.

[artiko88]

And another question, a bit off original topic, but since we are touching finer grain data i thought i'd better ask it here.

While in the examples hbt.elapse is used most of the time, there is hbt.wait_next_feed method that supposedly provides a possibility to use events as triggers. Looking at the code, eventually this method returns a response from goto method with a timestamp of next row in the data file which at the first glance should be the next event.

So i have a preprocessed file with 37M rows of data with trades and book updates. I am trying to calculate the number of events in it with a simple code:

    events = 0
    while hbt.wait_next_feed(False):
        events += 1
        hbt.clear_last_trades()
    print('events: ', events)

For some reason i only get ~3.3M events. I also tried calculating bbo updates using similar logic and only get 300K updates which i believe is a big underestimation.

So can hbt.wait_next_feed be used to obtain true event based backtesting or does it somehow skip some of the events? If it does, how does one make backtester to step further only one event at a time?

Edit:
Found the reason why it happens. The data i am using has microsecond precision and there are a lot of events with the same timestamp. So when processing the data, hftbacktest processes as many rows as necessary to go to next timestamp.