[roq-ftx] Validate MarketByPrice

[thraneh]

The online documentation is subtle, it says (about a checksum)

[...] you have likely lost or mishandled a packet and should re-subscribe to receive the initial snapshot

Live testing shows that this could actually mean that the CDN has lost packets. And the reason for saying this is that the updates are disseminated over WebSocket which is using lossless TCP/IP.

The real problem here is that the checksum is very expensive, it is the CRC32 of a string concatenation of the top 100 bid/ask prices/sizes.

One can either cache all string representations or compute those on the fly. Two evils -- either bad for the caches or bad for CPU cycles. Either way, this is a computation that ideally should be done on every update.

Detecting lost updates should be followed by resubscription: #31

[thraneh]

A client suggestion is to only checksum validate on every N updates. This strategy can be combined with a check on choice or inverted prices.

The reason this will work is that most updates are likely to be close to best bid/ask and lost messages will therefore manifest as some price level not getting removed when the market trades through it. That can quickly be noticed as choice or inverted prices.

The checksum calculation will catch it a little while later if the lost messages only involved changing a size (for an existing price level) or if the price level was removed further down the book.

If N is a flag (>=0), it's up to the user to choose the balance between correctness and speed

• N==0 means never compute checksum, only rely on checking choice / inverted prices
• N==1 means always compute checksum
• N>1 means some balance between checking choice / inverted prices (cheap) and checksum calculation (expensive)

[thraneh]

This has required changes to roq-server and in turn to roq-api:

MarketByPrice (the engine behind maintaining a L2 order book) was previously internal to roq-server. It now needs to be directly accessible by the gateways. We will gradually make the MarketByPrice interface accessible starting with the use-case required by this issue. The interface is now made public in roq-api.

[thraneh]

FTX resubscription has been added. Currently only triggered by detecting choice/inverted prices.

[thraneh]

Some notes

CRC32 checksum based on Python's string representation (str(value))

We must match Python exactly (fmt can help a bit, but many workaround are required to match).

• The value zero is always "0.0"
• No trailing zeros after significant decimals, i.e. "0.0123" (not "0.01230")
• Any number smaller than 1e-4 is always shown using scientific notation, i.e. "1e-5" (not "0.00001")

This is work in progress and must be tested over a period before we can even begin optimizations..

Minimum increment size not respected

• Have seen a size of 0.05178 (5 decimal precision) when reference data tells us 3 decimal precision
• Have seen a price of 0.0155125 (7 decimal precision) when we should expect 6 decimal precision
• Have seen a price of 0.0412 and a tick size of 0.00025, the ratio is 164.8

Order book window is a "fuzzy" 100

Online documentation states

The orderbook channel provides data about the orderbook's best 100 orders on either side.

However, looking at FTX's official Python library, we see there's no truncation to 100: https://github.com/ftexchange/ftx/blob/85beb7d3cc452277d9c7b648633ee761f0152851/websocket/client.py#L120

For example, this update seems to be an update removing liquidity from second best ask and adding it to best ask (an order modification?)

{action=UPDATE, bids=[{price=39.095, size=174.1}, {price=35.5775, size=0}], asks=[{price=39.4925, size=48}, {price=39.495, size=0}], checksum=-775538821, time=1631189595116ms}

What we have here is that a price level (39.495) has to be removed from the ask side of the book, and another price level (39.4925) must be added or updated (as it was: updated).

If this was a proper update to an always 100 deep book, it would also include the 100th worst ask price as well (because it re-enters the window).

The conclusion is that we must keep everything that moved outside the window. This is problematic because order actions may happen outside the window and not communicated to the users -- anything outside top 100 is probably not to be trusted.

Required changes (experiments and requires more testing)

MarketByPrice configuration to include tick-size multiplier (1.0e-2), min-trade-vol override (1.0e-8) and unlimited max depth.

[thraneh]

More fun stuff

• Seeing min-trade-vol as 1000000 and the book contains a size of 3.2 (XRPBEAR/USDT)

[thraneh]

Summary of changes

• The order book must not be truncated to 100
• The raw tick_size and min_trade_vol can't be used to convert floating point values to integers (needed for efficient maintenance of the order book). It works if we use multipliers for the increments (1e-2 for tick_size and 1e-4 for min_trade_vol) and also cap this value when the increment is >1.
• These assumptions must be communicated by gateway to clients so they also can build a proper book. The GatewaySettings structure has been updated with the new requirements.
• The client::DepthBuilder structure did not handle GatewaySettings, that's now changed. Strategies must themselves apply GatewaySettings after each reconnect!
• The changes to client::DepthBuilder now inherits from market::MarketByPrice, the interface directly into the logic for maintaining the order book. The idea is that we should gradually get rid of client::DepthBuilder.

[thraneh]

The gateway is currently defaulting to not validate checksum. This can be enabled by --ws_order_book_checksum_every 1.

However, optimizations are required. As can be confirmed by running roq-ftx-benchmark, the first implementation is very costly

That's 70 microseconds for a 100 level deep order book...

[thraneh]

A new checksum calculator has been implemented which does not use the fmt library, it is 4x faster

[thraneh]

This seems to work.

As expected, processing times are much slower when validating

For reference, this is without validation