NIC bandwidth is finite. As we scale up the number of turnovers per second, we will exhaust that bandwidth with snapshots.
We will need to manage NIC usage explicitly. When we take a snapshot, we might have to keep it locally for a while. We need to do something like:
- Publish that we have snapshot X on this node
- When a request comes in for actor A which needs snapshot X:
- route it to a worker on a node which has snapshot X if possible
- If not possible, see if the data exists in the backing store (meaning any node can restore)
- If so pick any worker
- If not, park the request and retry
Depending on how fancy we need to get we might notify the storage on the node that does have the snapshot that it should move that upload to the front of the queue.
NIC bandwidth is finite. As we scale up the number of turnovers per second, we will exhaust that bandwidth with snapshots.
We will need to manage NIC usage explicitly. When we take a snapshot, we might have to keep it locally for a while. We need to do something like:
Depending on how fancy we need to get we might notify the storage on the node that does have the snapshot that it should move that upload to the front of the queue.