-
-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Async Filer Metadata Backup
Chris Lu edited this page Nov 27, 2021
·
6 revisions
It is desirable to have a copy of the filer metadata. However, it is not always easy to setup replications for filer metadata store:
- The store does support stream replication.
- The store is embedded leveldb, RocksDB, etc.
- It is just complicated.
We can continuously backup filer metadata to a different store, without running a filer instance.
Metadata Changes
Filer --------------------> `weed filer.meta.backup` ---> A backup store(LevelDB/RocksDB/
Mysql/Postgres/
Redis/Cassanra/...)
Just need to configure the backup store the same way as filer.toml
.
weed filer.meta.backup
can be stopped and resumed. The metadata backup progress is tracked in the backup store itself. So you can pause/resume any time.
It is worth noting that the backup store can be different from the source filer store. E.g., you can use a cheaper on disk LevelDB as a remote store to backup Redis.
Metadata Changes
Filer --------------------> `weed filer.meta.backup` ---> A backup store(LevelDB)
on Redis
You can even do this:
Metadata Changes sqlite streaming
Filer --------------------> `weed filer.meta.backup` ---> A backup store(Sqlite) ------------------> S3
on Redis
- Replication
- Store file with a Time To Live
- Failover Master Server
- Erasure coding for warm storage
- Server Startup Setup
- Environment Variables
- Filer Setup
- Directories and Files
- Data Structure for Large Files
- Filer Data Encryption
- Filer Commands and Operations
- Filer JWT Use
- Filer Cassandra Setup
- Filer Redis Setup
- Filer YugabyteDB Setup
- Super Large Directories
- Path-Specific Filer Store
- Choosing a Filer Store
- Customize Filer Store
- Migrate to Filer Store
- Add New Filer Store
- Filer Store Replication
- Filer Active Active cross cluster continuous synchronization
- Filer as a Key-Large-Value Store
- Path Specific Configuration
- Filer Change Data Capture
- Cloud Drive Benefits
- Cloud Drive Architecture
- Configure Remote Storage
- Mount Remote Storage
- Cache Remote Storage
- Cloud Drive Quick Setup
- Gateway to Remote Object Storage
- Amazon S3 API
- AWS CLI with SeaweedFS
- s3cmd with SeaweedFS
- rclone with SeaweedFS
- restic with SeaweedFS
- nodejs with Seaweed S3
- S3 API Benchmark
- S3 API FAQ
- S3 Bucket Quota
- S3 API Audit log
- S3 Nginx Proxy
- Hadoop Compatible File System
- run Spark on SeaweedFS
- run HBase on SeaweedFS
- run Presto on SeaweedFS
- Hadoop Benchmark
- HDFS via S3 connector
- Async Replication to another Filer [Deprecated]
- Async Backup
- Async Filer Metadata Backup
- Async Replication to Cloud [Deprecated]
- Kubernetes Backups and Recovery with K8up