Backing up the WAL

[diptanu]

Hi @drmingdrmer do you guys back up the WAL or the snapshot somewhere in datafuse ? I am trying to figure out how to make it easy for us to stream the WAL to S3 so that if we lose the nodes we can bootstrap the cluster without losing the source of truth of the data in the state machine

[github-actions]

👋 Thanks for opening this issue!

Get help or engage by:

• /help : to print help messages.
• /assignme : to assign this issue to you.

[drmingdrmer]

In databend, we just export all of the log and the snapshot to a file and backup it. The following is the server side export implementation:
https://github.com/datafuselabs/databend/blob/313b40ed3ff1cfe001bc6e2f806773d9001861d0/src/meta/service/src/store/store_inner.rs#L394

Fix the above url

[diptanu]

@drmingdrmer Do you do this in a continuous basis for databend or is this happens after every couple of hours?

[drmingdrmer]

@drmingdrmer Do you do this in a continuous basis for databend or is this happens after every couple of hours?

It is a cronjob and is executed by schedule. It is not a incremental backup :(

[Xuanwo]

Hi @drmingdrmer, would you be open to incorporating that logic into openraft? This way, users wouldn't have to replicate the same work.

[schreter]

Just my 2c: I don't think there is a generic way to back up logs. We'll also back up to S3, but that's just one of the options. We must support 5 different hyperscalers, all with different APIs. From openraft POV, the log is continuous and how you cut it for backup is your beer. In our case, once a log segment is filled, it can be backed up. Once openraft tells us to purge, it's removed locally (unless the backup didn't yet finish).