Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incremental snapshot support #8888

Closed
swapnilgm opened this issue Nov 16, 2017 · 10 comments
Closed

Incremental snapshot support #8888

swapnilgm opened this issue Nov 16, 2017 · 10 comments

Comments

@swapnilgm
Copy link
Contributor

I was wondering, is there any incremental snapshot support for etcd? I don't see any command on etcdctl for same as of now. For use cases where one requires regular snapshots of etcd for backup purpose, having same data as part of multiple full snapshots is really costly from storage point of view.
It would be nice to have this feature. Do you already have any plan for this?

As far as my investigation lead,

  • There is some incremental snapshot implementation used internally by etcd. But it dosen't expose this to client at all. There is no API or command to access this. Why is it so?
  • make-mirror command over etcdctl uses the snapshot/restore and watch based mechanism to sync the clusters. Why can't we use similar approach to have incremental snapshot? Here I propose, instead of keeping PUT and DELETE operation from watch events on destination client, we can dump it in json or some defined format in file say inc_snapshot_fromRev_toRev. So that at the time of restore, user can support backup directory with full snapshot and these incremental snapshot. And our restore command, with some additional flag to know that we have to restore from incremental snapshot list, will restore the etcd cluster. While restoring, it will restore cluster from base full snapshot. And then apply the operation from incremental snapshots sequentially using probably embedded etcd.
@swapnilgm
Copy link
Contributor Author

@xiang90, Maintainers: Any opinion/updates about this issue?

@xiang90
Copy link
Contributor

xiang90 commented Dec 16, 2017

etcd/raft does snapshot internally to reclaim disk from WAL files. We do not support incremental snapshot from external. But that is something doable and wont be super hard. Do you have any interest working on it?

@swapnilgm
Copy link
Contributor Author

Yeah sure. I'll take it forward.

@hexfusion
Copy link
Contributor

I think this is important for sane backup/restore strategy +1

@xiang90 xiang90 added this to the v3.5.0 milestone Jan 24, 2018
@gaddamas
Copy link

gaddamas commented Aug 24, 2018

@xiang90, Could you please point me to documentation on snapshots or explain in detail about how snapshot functionality is used in etcd. Some questions: 1) is there snapshot at raft log (wal) level and another snapshot at bbolt db level. after adding 900k keys to a 3 node cluster, i have restarted all three nodes. two of the nodes after restart were busy with exchanging snapshot as shown in the log below. Is one node trying to send entire db to another node or is it trying to send snapshot of log from last committed index 900168? Thank you
sender node:
2018-08-23 15:34:24.489264 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:24.489504 I | rafthttp: start to send database snapshot [index: 900167, to f078dbaf5ed2490b]...
2018-08-23 15:34:24.491349 W | rafthttp: database snapshot [index: 900167, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
...
2018-08-23 15:34:25.090355 I | etcdserver: wrote database snapshot out [total bytes: 9223372036854775807]
2018-08-23 15:34:25.090482 I | rafthttp: start to send database snapshot [index: 900168, to f078dbaf5ed2490b]...
2018-08-23 15:34:25.092223 W | rafthttp: database snapshot [index: 900168, to: f078dbaf5ed2490b] failed to be sent out (ioutil: short read)
receiver node:
2018-08-23 15:34:24.392607 I | rafthttp: receiving database snapshot [index:900167, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.392890 E | rafthttp: failed to save KV snapshot (unexpected EOF)
2018-08-23 15:34:24.489257 E | etcdserver: publish error: etcdserver: request timed out
2018-08-23 15:34:24.690707 I | rafthttp: receiving database snapshot [index:900168, from ec5c70c3e0fa567c] ...
2018-08-23 15:34:24.691197 E | rafthttp: failed to save KV snapshot (unexpected EOF)
.....

@wenjiaswe
Copy link
Contributor

cc @jingyih backup/restore
cc @jpbetz nonvoting members

@jpbetz
Copy link
Contributor

jpbetz commented Oct 2, 2018

Part of the non-voting member design describes non-disruptive snapshot transmission (#9161). Once that's in place, we could just enable it for etcdctl snapshot save?

@xiang90
Copy link
Contributor

xiang90 commented Oct 2, 2018

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

@moooofly
Copy link

etcdctl snapshot save

etcdctl snapshot save --forever could potentially starts a non-voting member and keep on receiving data and making snapshots.

find no --forever flag in etcdctl snapshot save command

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl version
etcdctl version: 3.3.11
API version: 3.3
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 
[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# etcdctl snapshot save -h
NAME:
        snapshot save - Stores an etcd node backend snapshot to a given file

USAGE:
        etcdctl snapshot save <filename>

GLOBAL OPTIONS:
      --cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle
      --cert=""                                 identify secure client using this TLS certificate file
      --command-timeout=5s                      timeout for short running command (excluding dial timeout)
      --debug[=false]                           enable client-side debug logging
      --dial-timeout=2s                         dial timeout for client connections
  -d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints
      --endpoints=[127.0.0.1:2379]              gRPC endpoints
      --hex[=false]                             print byte strings as hex encoded strings
      --insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints
      --insecure-skip-tls-verify[=false]        skip server certificate verification
      --insecure-transport[=true]               disable transport security for client connections
      --keepalive-time=2s                       keepalive time for client connections
      --keepalive-timeout=6s                    keepalive timeout for client connections
      --key=""                                  identify secure client using this TLS key file
      --user=""                                 username[:password] for authentication (prompt if password is not supplied)
  -w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table)

[root@e5d68e04-f6f6-11e9-b1f2-00163e069dac ~]# 

@stale
Copy link

stale bot commented Apr 6, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 6, 2020
@stale stale bot closed this as completed Apr 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

8 participants