Backup

Although the processor is fully able to cope with crashes and connection lose with the database and even with some data loss (which is why we 'SET synchronous_commit TO off;' in the database setup script for unimportant data), the loss of actual blocks is a huge problem. A new block may be constructed that differs from the lost block, which means any nodes will now reject any future blocks as the chain of blocks has been modified. Hence there need to be fully up to date backups at all times. For this two backup solutions are recommended:

The first one is synchronous replication of the postgres database. This is preferred as it will ensure minimal to no downtime in case of crashes and can also be used for load balancing. Using the default setup the processor will only ensure the blocks are replicated synchronous, while other (unimportant) data is replicated asynchronous to ensure high throughput. Note that if you wish to use load balancing the processor assumes that insignificant data it reads is up to date. If the load balancing software cannot guarantee this "SET synchronous_commit TO remote_apply;". Should the database crash the backup can take its place and no data loss will occur while having minimal downtime.

The second option is to run a node (or even use someone else's node IF you trust that node) as all nodes receive a full copy of the blockchain. To reduce downtime have the node running at all times, rather then starting it when needing to restore from a backup. If you use this method and need to do a restore: shutdown the processor if it is still running. Should the node (of the processor, not the backup node) have gone down start it again with with a high VNODE_BLOCKINTERVAL and a high VNODE_LATESTEXISTINGBLOCK to give the backup node a chance to download and process the entire blockchain including the last few blocks. After the backup node has received and processed the latest blocks that were mined (you can see how far it is with processing using "SELECT value FROM info WHERE key = 'nextBlock';", which should match the latest block id + 1), eighter shutdown the backup node or make a database dump of that nodes database and start a new database from that dump. Make sure you run NodeToProcessor.sql on the database and start the processor. (And possibly restart the node of the processor with normal params again.) Note that if the backup node was unable to retrieve all blocks in the blockchain, any nodes that have more blocks will likely consider the blockchain invalid from now on. In addition any transactions that were delivered to the processor, but not yet put into a block will certainly be lost. Hence the first backup strategy is recommended.

Do not try to use an old version of the processor database in order to make the backup node catch up faster, it is not made for this!