Skip to content

Backup metastore#820

Merged
MarinPostma merged 10 commits intomainfrom
durable-meta-store
Jan 4, 2024
Merged

Backup metastore#820
MarinPostma merged 10 commits intomainfrom
durable-meta-store

Conversation

@MarinPostma
Copy link
Copy Markdown
Contributor

@MarinPostma MarinPostma commented Dec 21, 2023

This PR implements the meta store backup and restore logic using a bottomless-enabled connection. I cleaned up the code a bit to to reuse WAL wrapper patterns by implementing WrapWal for Option<T>
and used that for ReplicationWal and the meta store connection.

Configuration for the meta store durability is passed at thanks to new cli arguments:

      --backup-meta-store
          Enable backup for the metadata store

      --meta-store-access-key-id <META_STORE_ACCESS_KEY_ID>
          S3 access key ID for the meta store backup

      --meta-store-secret-access-key <META_STORE_SECRET_ACCESS_KEY>
          S3 secret access key for the meta store backup

      --meta-store-region <META_STORE_REGION>
          S3 region for the metastore backup

      --meta-store-backup-id <META_STORE_BACKUP_ID>
          Id for the meta store backup

      --meta-store-bucket-name <META_STORE_BUCKET_NAME>
          S3 bucket name for the meta store backup

      --meta-store-backup-interval-s <META_STORE_BACKUP_INTERVAL_S>
          Interval at which to perform backups of the meta store

      --meta-store-bucket-endpoint <META_STORE_BUCKET_ENDPOINT>
          S3 endpoint for the meta store backups

@MarinPostma MarinPostma marked this pull request as ready for review December 21, 2023 17:40
Ok(ret)
}

fn exclusive_mode(&mut self, op: c_int) -> Result<()> {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we going to have issues with these not being forwarded to the inner wal impl?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's forwarded by default; notice how we are not implementing Wal but WrapWal now :) this is a convenient trait that helps compose WAL implementations. Upstream, I have stuff like:

        let wal_manager = Sqlite3WalManager::default()
            .wrap(TakeReaderWrapper(tmp.path().to_path_buf()))
            .wrap(ReplicationIndexInjectorWrapper);

so that it's super easy to only implement what you need for WAL implementation that wrap other WAL implementations

Comment on lines +3 to +5
use std::collections::HashMap;
use std::path::Path;
use std::sync::Arc;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😯

@LucioFranco
Copy link
Copy Markdown
Contributor

I think you're still missing the instrument macro though I think we should add that.

@MarinPostma MarinPostma enabled auto-merge January 4, 2024 09:54
@MarinPostma MarinPostma added this pull request to the merge queue Jan 4, 2024
Merged via the queue into main with commit 40119d8 Jan 4, 2024
@MarinPostma MarinPostma deleted the durable-meta-store branch January 4, 2024 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants