Buffered Writes to Disk

For a detailed description of why we buffer the writes to the database, see entry 6 in the change log on the Wiki.

The basic summary is that having fast, non-blocking, concurrent writes to disk is the holy grail. Disk is way slower than RAM, and adding a database in the mix makes things even worse because the RDBMS has to ensure there are no conflicts, it needs to build indexes, etc while ensuring some other client can connect and query for some data. With this in mind, it's obvious that the data being written to disk is important and needs to get there in one piece, but it's also clear that sometimes we need to have a fast way to access the data even as its being written to Postgres.

The evolution of disk writes in this project was pretty standard. First, we were completely in-memory, with no persistence. Then, to maximize consistency at the peril of performance, we wrote to postgres every time the exchange's state changed. This was brutally slow, so we started to buffer the data and write it to disk in batch transactions, which decreased latency for the end-user, but increased unavailability during the time our buffers were flushed to disk. Once Redis was integrated, we could eliminate that downtime by avoiding the database for queries all together, opting to read and write directly to Redis in the meantime (during normal exchange operations).