Problem with orderbook data ?

[Leobouloc]

I am using btfxwss to regularly fetch orderbook data. On the past twelve days, the connection has been reset about 50 times. At each reset, I receive a new snapshot, and notice that my version of the orderbook was wrong with values being shifted up or down both for the bid and ask values (but the gap always seems coherent)...

I believe I properly implemented the algorithm (code below) described in Bitfinex's API doc (https://docs.bitfinex.com/v2/reference#ws-public-order-books) and can't figure out what is wrong... Any ideas what could be the cause of this ?

Thank you for your help

class OrderBook(dict):
    def __init__(self, books_q):
        self.books_q = books_q
        
    def _init_from_snapshot(self, snapshot):
        """Create a new book from snapshot."""
        self['asks'] = dict()
        self['bids'] = dict()
        assert snapshot
        for (price, count, amount) in snapshot:
            self._apply_single_update(price, count, amount)
        
    def _apply_single_update(self, price, count, amount):
        if count > 0:
            if amount > 0:
                self['bids'][price] = (count, amount)
            else:
                self['asks'][price] = (count, -amount)      
        else:
            assert count == 0
            assert amount in [-1, 1]
            
            if amount == 1:
                del self['bids'][price]
            else:
                del self['asks'][price]               

    def get(self):
        """Update the book to the current state."""
        print('Has vals:', not self.books_q.empty())
        
        while not self.books_q.empty():
            vals, self.timestamp = self.books_q.get()
            assert len(vals) == 1
            
            # Differentiate between snapshot and update
            if len(vals[0]) == 3:
                (price, count, amount) = vals[0]
                self._apply_single_update(price, count, amount)
            else:
                self._init_from_snapshot(vals[0])

I use the class above as follows:

[... initialize wss ...]
pair = 'BTCUSD'
PRECISION = 'P0'
FREQUENCY = 'F1' # F0 for realtime, F1 for 2sec
LENGTH = '100' # 25 or 100

WRITE_INTERVAL = 20 # in seconds  

wss.subscribe_to_order_book(pair, prec=PRECISION, freq=FREQUENCY, len=LENGTH)

# Create book object
book = OrderBook(wss.books(pair))

# Get snapshot
book.get()

last_write = 0
while True:
    # Fetch and write data at a regular interval
    if time.time() - last_write >= WRITE_INTERVAL: 
        # Update (or get snapshot if connection was reset)
        book.get()
        last_write = time.time()
        [... write book as json ...]

[deepbrook]

Hey @Leobouloc ,

btfxwss does not handle mainting an orderbook. It only stores the messages you need to process. This means, that you only ever process a single entry update when you call book.get - instead, you should call book.get() in a separate thread in a while True loop, in order to keep it up to date OR process all data in the queue (although this may or may not finish, depending on how fast messages come in).

[Leobouloc]

Hi, I am not sure which part I got wrong. Shouldn't the code below (the code that is executed when I call book.get) cover what you are saying ?

    def get(self):
        """Update the book to the current state."""
        print('Has vals:', not self.books_q.empty())
        
        while not self.books_q.empty():
            vals, self.timestamp = self.books_q.get()
            assert len(vals) == 1
            
            # Differentiate between snapshot and update
            if len(vals[0]) == 3:
                (price, count, amount) = vals[0]
                self._apply_single_update(price, count, amount)
            else:
                self._init_from_snapshot(vals[0])

(books_q is an object initiated by: wss.books(pair))

I believe what I am doing is: every 20 seconds, call get; get takes all the updates that were accumulated during the past 20 seconds (until sekf.books_q.empty()) and uses them to update the orderbook.

[deepbrook]

Ah, my bad.

This could be a race-condition - since you update the book only when you call get(), by the time it's done the order book may have changed already, hence you end up with different books when you received the snapshot.

Check if this also happens if you update the book continuously - it may still happen (updating continuously does not eliminate race conditions), but hopefully less often. If that's the case, you can take it from there and see how you can eliminate such a state in your code.

Does that help?

[Leobouloc]

Hi Nils, I don't believe this is a speed issue as my code always exits the while loop (I write only once I exit the while loop). Also the changes I observe (10% in the min(ask) or max(bids)) are not the type of changes that usually occur within 20 seconds (or at least not that frequently); I assume my issue is due to a slow drift in my orderbook but I can't figure out why :(

One surprising fact is that the orderbook seems to drift in a coherent manner between asks and bids since the gap between min(ask) or max(bids) stays coherent (rather small)...