Use reliable queues by Redis to process incoming orders

[hodlbirb]

By using RPOPLPUSH we can guarantee that the order will be processed exactly once:

1. The service subscribe (PSUBSCRIBE) on CONSUMER queue keyspace notifications

PSUBSCRIBE "__keyspace@0__:CONSUMER"

2. Waits until LPUSH keyevent is fired

1) "pmessage"
2) "__keyspace@0__:CONSUMER"
3) "__keyspace@0__:CONSUMER"
4) "lpush"  

3. Once it's triggered, the service pops the new order from the CONSUMER queue and atomically pushes to the CONSUMER_PROCESSING queue (RPOPLPUSH)

RPOPLPUSH "CONSUMER" "CONSUMER_PROCESSING"

4. Later, the state of the popped item is saved in the datastore with status processing
5. Upon successful persistence of the order the consumer removes (LREM) the element from the CONSUMER_PROCESSING queue

LREM "CONSUMER_PROCESSING" 0 "order_id: 200, market: EUR_USD"

Edge cases:

• if the matching engine crashed and failed to save the status after popping the order from the producer queue, on next recovery it loads all orders with processing status from the datastore and removes already saved orders from the consumer queue, after cleanup of consumer queue it proceeds operation as normal