TigerBeetle offers exceptional resistance to file corruption, but has no story for how to handle the loss of (e.g.) a single node's data. If, say, a fire destroys a node's disks, there should be a documented and safe way for users to replace that node. This will also help when nodes data files are corrupted beyond repair--for instance, when the superblock is irrecoverable.
There is an undocumented recovery procedure: users can run tigerbeetle format to initialize a fresh data file for the node. From there, the normal repair process can recover the entire node's data file. This often works correctly. However, it has (at least) a few unsafe edges.
First, reformatting effectively causes the node to forget everything it had done. If an operation op were committed by 2/3 nodes, VSR would normally guarantee that the operation is durable. However, reformatting one of those nodes leaves op present on only a single node. If the cluster went through a view change, it could use the 2/3 majority which lack op to move on--destroying the single remaining copy of op. Operations which were acknowledged to clients might be lost.
Second, there may be issues where a node reformats with a new version of TigerBeetle, but the cluster has not actually migrated to that version yet. TigerBeetle could panic in replica.zig/open or replica.zig/sync_superblock_update_finish.
TigerBeetle offers exceptional resistance to file corruption, but has no story for how to handle the loss of (e.g.) a single node's data. If, say, a fire destroys a node's disks, there should be a documented and safe way for users to replace that node. This will also help when nodes data files are corrupted beyond repair--for instance, when the superblock is irrecoverable.
There is an undocumented recovery procedure: users can run
tigerbeetle formatto initialize a fresh data file for the node. From there, the normal repair process can recover the entire node's data file. This often works correctly. However, it has (at least) a few unsafe edges.First, reformatting effectively causes the node to forget everything it had done. If an operation
opwere committed by 2/3 nodes, VSR would normally guarantee that the operation is durable. However, reformatting one of those nodes leavesoppresent on only a single node. If the cluster went through a view change, it could use the 2/3 majority which lackopto move on--destroying the single remaining copy ofop. Operations which were acknowledged to clients might be lost.Second, there may be issues where a node reformats with a new version of TigerBeetle, but the cluster has not actually migrated to that version yet. TigerBeetle could panic in
replica.zig/openorreplica.zig/sync_superblock_update_finish.