Buffered Writes to Disk

For a detailed description of why we buffer the writes to the database, see entry 6 in the change log on the Wiki.

The basic summary is that having fast, non-blocking, concurrent writes to disk is the holy grail. Disk is way slower than RAM, and adding a database in the mix makes things even worse because the RDBMS has to ensure there are no conflicts, it needs to build indexes, etc while ensuring some other client can connect and query for some data. With this in mind, it's obvious that the data being written to disk is important and needs to get there in one piece, but it's also clear that sometimes we need to have a fast way to access the data even as its being written to Postgres.

The evolution of disk writes in this project was pretty standard. First, we were completely in-memory, with no persistence. Then, to maximize consistency at the peril of performance, we wrote to postgres every time the exchange's state changed. This was brutally slow, so we started to buffer the data and write it to disk in batch transactions, which decreased latency for the end-user, but increased unavailability during the time our buffers were flushed to disk. Once Redis was integrated, we could eliminate that downtime by performing the database queries asynchronously, opting to read and write directly to Redis in the meantime (during normal exchange operations).

The Buffer Structures

The buffers themselves are extremely simple queues. The Order buffer is a bit more complicated, so we'll briefly cover it here, and explain how we managed to decrease memory usage significantly.

Below is the basic structure of the OrderBuffer.

struct OrderBuffer {
    data: HashMap<i32, DatabaseReadyOrder>,
    state: BufferState
}

Before explaining it, we'll briefly look at its components, starting with the DatabaseReadyOrder.

struct DatabaseReadyOrder {
    pub action:       Option<String>,
    pub symbol:       Option<String>,
    pub quantity:     Option<i32>,
    pub filled:       Option<i32>,
    pub price:        Option<f64>,
    pub order_id:     Option<i32>,
    pub status:       Option<OrderStatus>,
    pub user_id:      Option<i32>,
    pub time_placed:  Option<DateTime<Utc>>,
    pub time_updated: Option<DateTime<Utc>>,
}

The first thing you may notice is that every field is an Option, the next thing you might notice is that it's nearly identical to an Order. The reason for the Options is pretty straightforward: there are situations where most of the data can be represented as None (1 byte).

Here, we make the distinction between known and unknown orders.

• Known: An order that Redis or the Database have seen in the past.
• Unknown: An order that is only known to the application, i.e it was seen for the first time recently.

Unknown orders will have Some(val) for most fields in their DatabaseReadyOrder, since we do not have this data stored in Redis or Postgresql. Known orders are different, since we're really just going to be updating a few fields like the status, filled, or time_updated. Since most fields for known orders never change, we just leave them as None, giving us a considerable space saving both in our buffers, and resulting in less computation when we generate SQL queries.

Buffer State

Next, we'll take a look at the states a buffer can be in.

enum BufferState {
    EMPTY,
    NONEMPTY,
    FULL,
    FORCEFLUSH, // set when some other module requests the buffers be flushed.
}

These states speak for themselves, we flush the buffers whenever they're FULL or have been forced to flush (ex. on shutdown).

OrderBuffer

Back to the OrderBuffer then,

struct OrderBuffer {
    data: HashMap<i32, DatabaseReadyOrder>,
    state: BufferState
}

Take note of the fact that we're using a HashMap rather than a Vec or some other queue like structure. This is another space-saving mechanism whereby we only store 1 instance of an order and modify it during program execution. This approach lets us flush the final DIFF rather than flushing several "events" that modified the order.