Implement redis-compatible (RDB) snapshotting support #293
Comments
danielealbano
added a commit
that referenced
this issue
Apr 9, 2023
This PR implements the necessary support to generate the RDB snapshots
in the background.
A new fiber has been introduced which takes care of generating a
snapshot, depending on the settings. A new set of parameters have been
introduced, example below with explanation in line.
```
snapshots:
# The path where the snapshot file will be stored, if the rotation is enabled, the path will be used as prefix and
# the timestamp of the start of the snapshot will be appended to the file name.
path: /var/lib/cachegrand/dump.rdb
# The interval between the snapshots, the allowed units are s, m, h, if not specified the default is seconds.
interval: 5m
# The number of keys that must be changed before a snapshot is taken, 0 means no limit
min_keys_changed: 1000
# The amount of data that must be changed before a snapshot is taken, the allowed units are b, k, m, g, if not
min_data_changed: 100mb
# Rotation settings, optional, if missing the snapshots rotation will be disabled
rotation:
# The max number of snapshots files to keep, minimum 2
max_files: 10
```
The example is from cachegrand.yaml.skel
The new mechanism takes care of reporting every 3 seconds a status
update.
Closes #293
Notes:
- Currently the redis commands SAVE and BGSAVE are not implemented, so
it's not possible to trigger a backup on demand (#314)
- The SHUTDOWN command needs to be updated to support SAVE and NOSAVE
(#315)
- The shutdown logic needs to be updated to trigger a dump at the
shutdown unless the SHUTDOWN NOSAVE command has been issued (#316)
- The current implementation also doesn't compress strings with the LZF
algorithm as liblzf is causing segfault and the issue has to be
investigated (#312)
- Tests for the high level snapshotting process (implemented in
storage_db_snapshot.c mostly) are not included in this PR (#317)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To both be able to save and load the Key-Value database when the in-memory backend is used and to be compatible with Redis and be able to import its snapshots, as well as allow Redis to load cachegrands' snapshots, it's necessary to implement a snapshotting mechanism that produces RDB compatible files.
Plenty of well written documentation can be found at https://rdb.fnordig.de/ ( https://rdb.fnordig.de/file_format.html ), although it doesn't include the latest changes from version 8 and 9 it's pretty well organized and it's a good reference.
Additional documentation for the newest versions can be found here
https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/docs/RDB_Version_History.textile
https://github.com/sripathikrishnan/redis-rdb-tools/blob/master/docs/RDB_File_Format.textile
Tools in https://rdb.fnordig.de/ and https://github.com/sripathikrishnan/redis-rdb-tools/ can also be used for further validation of the file format compatibility.
The text was updated successfully, but these errors were encountered: