FEAT: database persistence to disk

[edwinkys]

Use case

Currently, the database data, key-value pairs and index, is stored only in memory. This means that every time the database is restarted, we need to re-add the key-value and rebuild the index.

This feature should allow the database data to be persisted to disk so that when there is crash or shutdown, the data and index of previous session can still be used.

Proposed solution

The database should store a snapshot of the key-value pairs periodically. Every time, the database is started, the database will lookup the most recent snapshot and prepopulate the key-value store and rebuild the index based on the existing key-value pairs.

Additional Context

• https://crates.io/crates/bincode

[edwinkys]

Revision on proposed solution

In addition to creating periodical snapshots, we should also implement append-only logging that will record write/delete operations.

This allows us to increase the durability of the persistence process, increase intervals between snapshots, and decrease the size of the log file.