Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

New commands: PIPESAVE and DUMPSAVE #1185

Open
wants to merge 3 commits into from

5 participants

@mpalmer

I can't believe I haven't submitted this before...

This patch implements a pair of commands to make RDB backups easier, without chewing up IOPS on the system unnecessarily. Both commands emit an RDB file over a socket, rather than writing them to disk. The differences are:

  • PIPESAVE invokes a system command (as specified by the new config variable pipesavecommand) and writes the RDB to that command's stdin. Why would you want this? Well, you can implement a pipeline to compress, encrypt, and send-to-S3 an RDB file, all without it touching disk. This can, for example, reduce the time to do a backup from 3 hours to 15 minutes.
  • DUMPSAVE writes the RDB file straight back at the client who called the command. The client should be grabbing that data and doing something with it (writing it to disk would be a good recommendation). The purpose of this command is to allow a backup server to request an RDB file, without needing to either (a) become a slave (with the associated memory usage), or (b) Have the RDB written to disk locally.
mpalmer and others added some commits
@mpalmer mpalmer New command: pipesave
Allows you to specify a command to run (via the config file *only*, to
prevent people from running arbitrary code on your machine via redis), which
will be invoked and fed the contents of an RDB dump file.

Useful for taking offsite backups (say, to S3 or a centralised backup server)
or a periodic replication mechanism when you don't want (or don't have the
available memory) to run twice as many redis instances everywhere.

Redis treats these children as it would treat a child forked to BGSAVE.
This simplifies the code (rdb_child_pid is already checked where it matters),
but means you can't run a BGSAVE and PIPESAVE concurrently.
ef6319f
@saj saj New command: dumpsave
Allows a client to connect and request an in-band RDB dump down the
connection.

This is ideal for automated backup systems, as it lets the backup
server "pull" a consistent copy of the Redis data instead of
hoping that a suitable RDB file will sitting on disk, ready for
pickup.

Note that this behaviour does not conform to the Redis protocol
specification in any way.
117cacf
@saj saj Provide a sample client for making use of the DUMPSAVE feature
The DUMPSAVE command's response does not follow the Redis protocol
specification. Instead of a standard Redis bulk reply, Redis will
send raw RDB data down the line without warning.

Never use a regular Redis client library to submit a DUMPSAVE. That
said, the DUMPSAVE exchange is so simple that it requires almost
no effort at all to implement a working client yourself.
2c1e87c
@mezzatto

PIPESAVE with the S3 backup use case seems pretty usefull to me.

@charl

Would these commands still have the RDB requirement where when the dump is kicked off it may use up 2-3 time as much RAM as the current redis db is using?

@georgepsarakis

I think PIPESAVE can be implemented with named pipes (probably), NFS mount or SSHFS. DUMPSAVE would introduce issues such as network stability since you are receiving data in a compressed, binary format and I am guessing it would be very hard to verify the integrity of the received data. The client already receives data in AOF format so that would be a more sane alternative although somewhat slower and resulting in larger output (but again you do not need an extra command for this).

Maybe what you need is simply I/O throttling in the RDB dump process, an extra parameter limiting the IOPS during SAVE/BGSAVE could be added? That could make sense for every use case, not only remote backups.

@mpalmer

@charl: RDB doesn't have a requirement to use three times the RAM of the master redis process, so no, these commands don't either. They do, however, fork a child, so the same CoW issues will exist. However, since the dump will probably stream faster over the network than to disk, the maximum overhead should be less.

@georgepsarakis: You could potentially use a named pipe, but it'd be messy to implement, and would still require a change to Redis to allow it to dump an RDB to a file other than dbfilename (because if we overrode dbfilename to be the named pipe, every time you shut down Redis all your data would go down the pipe and there'd be nothing to read on startup). I have no idea how you'd use an NFS mount or SSHFS to encrypt an RDB and send it to Amazon S3 without touching local disk. I also can't comprehend how dumping an RDB over a network introduces any more network instability than copying it over that same network after writing it to local disk. I'm not sure how to even address your suggestion of AOF as an alternative. It isn't. As for I/O throtting... I want my backups to complete faster, not even slower.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jul 7, 2013
  1. @mpalmer

    New command: pipesave

    mpalmer authored
    Allows you to specify a command to run (via the config file *only*, to
    prevent people from running arbitrary code on your machine via redis), which
    will be invoked and fed the contents of an RDB dump file.
    
    Useful for taking offsite backups (say, to S3 or a centralised backup server)
    or a periodic replication mechanism when you don't want (or don't have the
    available memory) to run twice as many redis instances everywhere.
    
    Redis treats these children as it would treat a child forked to BGSAVE.
    This simplifies the code (rdb_child_pid is already checked where it matters),
    but means you can't run a BGSAVE and PIPESAVE concurrently.
  2. @saj @mpalmer

    New command: dumpsave

    saj authored mpalmer committed
    Allows a client to connect and request an in-band RDB dump down the
    connection.
    
    This is ideal for automated backup systems, as it lets the backup
    server "pull" a consistent copy of the Redis data instead of
    hoping that a suitable RDB file will sitting on disk, ready for
    pickup.
    
    Note that this behaviour does not conform to the Redis protocol
    specification in any way.
  3. @saj @mpalmer

    Provide a sample client for making use of the DUMPSAVE feature

    saj authored mpalmer committed
    The DUMPSAVE command's response does not follow the Redis protocol
    specification. Instead of a standard Redis bulk reply, Redis will
    send raw RDB data down the line without warning.
    
    Never use a regular Redis client library to submit a DUMPSAVE. That
    said, the DUMPSAVE exchange is so simple that it requires almost
    no effort at all to implement a working client yourself.
This page is out of date. Refresh to see the latest.
View
8 redis.conf
@@ -142,6 +142,14 @@ rdbchecksum yes
# The filename where to dump the DB
dbfilename dump.rdb
+# When a client requests PIPESAVE, Redis will fork, popen(3) whatever
+# you have defined in pipesavecommand, and write an RDB dump to it.
+# For security reasons, this configuration knob must be supplied in
+# this file (if it is supplied at all) and cannot be overriden at
+# runtime with CONFIG SET.
+#
+# pipesavecommand "/usr/bin/whatever \"blah blah\""
+
# The working directory.
#
# The DB will be written inside this directory, with the filename specified
View
8 src/config.c
@@ -373,6 +373,9 @@ void loadServerConfigFromString(char *config) {
server.hash_max_ziplist_entries = memtoll(argv[1], NULL);
} else if (!strcasecmp(argv[0],"hash-max-ziplist-value") && argc == 2) {
server.hash_max_ziplist_value = memtoll(argv[1], NULL);
+ } else if (!strcasecmp(argv[0],"pipesavecommand") && argc == 2) {
+ zfree(server.pipesavecommand);
+ server.pipesavecommand = zstrdup(argv[1]);
} else if (!strcasecmp(argv[0],"list-max-ziplist-entries") && argc == 2){
server.list_max_ziplist_entries = memtoll(argv[1], NULL);
} else if (!strcasecmp(argv[0],"list-max-ziplist-value") && argc == 2) {
@@ -1005,6 +1008,11 @@ void configGetCommand(redisClient *c) {
addReplyBulkCString(c,buf);
matches++;
}
+ if (stringmatch(pattern,"pipesavecommand",0)) {
+ addReplyBulkCString(c,"pipesavecommand");
+ addReplyBulkCString(c,server.pipesavecommand);
+ matches++;
+ }
if (stringmatch(pattern,"maxmemory-policy",0)) {
char *s;
View
5 src/config.h
@@ -105,6 +105,11 @@
#define rdb_fsync_range(fd,off,size) fsync(fd)
#endif
+/* Does your popen(3) support the 'e' flag? */
+#ifdef __linux__
+#define HAVE_POPEN_MODE_E 1
+#endif
+
/* Check if we can use setproctitle().
* BSD systems have support for it, we provide an implementation for
* Linux and osx. */
View
16 src/db.c
@@ -372,6 +372,22 @@ void shutdownCommand(redisClient *c) {
addReplyError(c,"Errors trying to SHUTDOWN. Check logs.");
}
+void pipesaveCommand(redisClient *c) {
+ if (rdbPipesaveBackground(server.pipesavecommand) == REDIS_OK) {
+ addReplyStatus(c,"Background pipesave started");
+ } else {
+ addReply(c,shared.err);
+ }
+}
+
+void dumpsaveCommand(redisClient *c) {
+ if (rdbDumpsaveBackground(c->fd) == REDIS_OK) {
+ addReplyStatus(c,"Background dumpsave started");
+ } else {
+ addReply(c,shared.err);
+ }
+}
+
void renameGenericCommand(redisClient *c, int nx) {
robj *o;
long long expire;
View
174 src/rdb.c
@@ -626,26 +626,84 @@ int rdbSaveKeyValuePair(rio *rdb, robj *key, robj *val,
return 1;
}
-/* Save the DB on disk. Return REDIS_ERR on error, REDIS_OK on success */
+/* Save the DB to a file. If filename begins with a pipe (`|'), then
+ * exec whatever program comes after it and write the DB to its stdin.
+ *
+ * Return REDIS_ERR on error, REDIS_OK on success. */
int rdbSave(char *filename) {
+ FILE *fp;
+ char tmpfile[256];
+
+ /* Open */
+ if (filename[0] == '|') {
+#ifdef HAVE_POPEN_MODE_E
+ fp = popen(filename+1, "we");
+#else
+ fp = popen(filename+1, "w");
+#endif
+ if (!fp) {
+ redisLog(REDIS_WARNING, "Failed saving the DB via pipe: %s", strerror(errno));
+ return REDIS_ERR;
+ }
+ } else {
+ snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
+ fp = fopen(tmpfile,"w");
+ if (!fp) {
+ redisLog(REDIS_WARNING, "Failed opening .rdb for saving: %s",
+ strerror(errno));
+ return REDIS_ERR;
+ }
+ }
+
+ /* Write */
+ if (rdbSaveToFileDescriptor(fp) == REDIS_ERR) {
+ if (filename[0] == '|') {
+ pclose(fp);
+ redisLog(REDIS_WARNING,"Write error saving DB to pipe: %s", strerror(errno));
+ } else {
+ fclose(fp);
+ unlink(tmpfile);
+ redisLog(REDIS_WARNING,"Write error saving DB on disk: %s", strerror(errno));
+ }
+ return REDIS_ERR;
+ }
+
+ /* Close */
+ if (filename[0] == '|') {
+ if (pclose(fp) < 0) {
+ redisLog(REDIS_WARNING, "Save to pipe failed: %s", strerror(errno));
+ return REDIS_ERR;
+ }
+ redisLog(REDIS_NOTICE,"DB saved to command '%s'", filename+1);
+ } else {
+ fclose(fp);
+
+ /* Use RENAME to make sure the DB file is changed atomically only
+ * if the generate DB file is ok. */
+ if (rename(tmpfile,filename) == -1) {
+ redisLog(REDIS_WARNING,"Error moving temp DB file on the final destination: %s", strerror(errno));
+ unlink(tmpfile);
+ return REDIS_ERR;
+ }
+ redisLog(REDIS_NOTICE,"DB saved on disk");
+ }
+ server.dirty = 0;
+ server.lastsave = time(NULL);
+ server.lastbgsave_status = REDIS_OK;
+ return REDIS_OK;
+}
+
+/* Save the DB to the given FILE pointer. We are responsible neither
+ * for opening nor closing the descriptor. */
+int rdbSaveToFileDescriptor(FILE *fp) {
dictIterator *di = NULL;
dictEntry *de;
- char tmpfile[256];
char magic[10];
int j;
long long now = mstime();
- FILE *fp;
rio rdb;
uint64_t cksum;
- snprintf(tmpfile,256,"temp-%d.rdb", (int) getpid());
- fp = fopen(tmpfile,"w");
- if (!fp) {
- redisLog(REDIS_WARNING, "Failed opening .rdb for saving: %s",
- strerror(errno));
- return REDIS_ERR;
- }
-
rioInitWithFile(&rdb,fp);
if (server.rdb_checksum)
rdb.update_cksum = rioGenericUpdateChecksum;
@@ -658,7 +716,6 @@ int rdbSave(char *filename) {
if (dictSize(d) == 0) continue;
di = dictGetSafeIterator(d);
if (!di) {
- fclose(fp);
return REDIS_ERR;
}
@@ -671,7 +728,7 @@ int rdbSave(char *filename) {
sds keystr = dictGetKey(de);
robj key, *o = dictGetVal(de);
long long expire;
-
+
initStaticStringObject(key,keystr);
expire = getExpire(db,&key);
if (rdbSaveKeyValuePair(&rdb,&key,o,expire,now) == -1) goto werr;
@@ -692,25 +749,9 @@ int rdbSave(char *filename) {
/* Make sure data will not remain on the OS's output buffers */
fflush(fp);
fsync(fileno(fp));
- fclose(fp);
-
- /* Use RENAME to make sure the DB file is changed atomically only
- * if the generate DB file is ok. */
- if (rename(tmpfile,filename) == -1) {
- redisLog(REDIS_WARNING,"Error moving temp DB file on the final destination: %s", strerror(errno));
- unlink(tmpfile);
- return REDIS_ERR;
- }
- redisLog(REDIS_NOTICE,"DB saved on disk");
- server.dirty = 0;
- server.lastsave = time(NULL);
- server.lastbgsave_status = REDIS_OK;
return REDIS_OK;
werr:
- fclose(fp);
- unlink(tmpfile);
- redisLog(REDIS_WARNING,"Write error saving DB on disk: %s", strerror(errno));
if (di) dictReleaseIterator(di);
return REDIS_ERR;
}
@@ -747,7 +788,7 @@ int rdbSaveBackground(char *filename) {
server.stat_fork_time = ustime()-start;
if (childpid == -1) {
redisLog(REDIS_WARNING,"Can't save in background: fork: %s",
- strerror(errno));
+ strerror(errno));
return REDIS_ERR;
}
redisLog(REDIS_NOTICE,"Background saving started by pid %d",childpid);
@@ -759,6 +800,79 @@ int rdbSaveBackground(char *filename) {
return REDIS_OK; /* unreached */
}
+int rdbPipesaveBackground(char *command) {
+ pid_t childpid;
+ char *pipecommand;
+
+ if (server.rdb_child_pid != -1) return REDIS_ERR;
+
+ if (!command || strlen(command) < 1) {
+ redisLog(REDIS_WARNING,"Cannot PIPESAVE without a configured pipesavecommand");
+ return REDIS_ERR;
+ }
+
+ if ((childpid = fork()) == 0) {
+ int retval;
+
+ /* Child */
+ if (server.ipfd > 0) close(server.ipfd);
+ if (server.sofd > 0) close(server.sofd);
+ pipecommand = zmalloc(strlen(command) + 2);
+ snprintf(pipecommand, strlen(command) + 2, "|%s", command);
+ retval = rdbSave(pipecommand);
+ exitFromChild((retval == REDIS_OK) ? 0 : 1);
+ } else {
+ /* Parent */
+ if (childpid == -1) {
+ redisLog(REDIS_WARNING,"Can't pipesave: fork: %s",
+ strerror(errno));
+ return REDIS_ERR;
+ }
+ redisLog(REDIS_NOTICE,"Background pipesave started by pid %d",childpid);
+ server.rdb_child_pid = childpid;
+ updateDictResizePolicy();
+ return REDIS_OK;
+ }
+ return REDIS_OK; /* unreached */
+}
+
+int rdbDumpsaveBackground(int fd) {
+ pid_t childpid;
+ FILE *fp;
+
+ if (server.rdb_child_pid != -1) return REDIS_ERR;
+
+ if ((childpid = fork()) == 0) {
+ int retval;
+
+ /* Child */
+ if (server.ipfd > 0) close(server.ipfd);
+ if (server.sofd > 0) close(server.sofd);
+ fp = fdopen(fd, "w");
+ if (!fp) {
+ redisLog(REDIS_WARNING, "Can't dumpsave: fdopen: %s",
+ strerror(errno));
+ exitFromChild(1);
+ }
+ retval = rdbSaveToFileDescriptor(fp);
+ exitFromChild((retval == REDIS_OK) ? 0 : 1);
+ } else {
+ /* Parent */
+ if (childpid == -1) {
+ redisLog(REDIS_WARNING,"Can't dumpsave: fork: %s",
+ strerror(errno));
+ return REDIS_ERR;
+ }
+ aeDeleteFileEvent(server.el,fd,AE_READABLE);
+ close(fd);
+ redisLog(REDIS_NOTICE,"Background dumpsave started by pid %d",childpid);
+ server.rdb_child_pid = childpid;
+ updateDictResizePolicy();
+ return REDIS_OK;
+ }
+ return REDIS_OK; /* unreached */
+}
+
void rdbRemoveTempFile(pid_t childpid) {
char tmpfile[256];
View
3  src/rdb.h
@@ -101,8 +101,11 @@ int rdbSaveObjectType(rio *rdb, robj *o);
int rdbLoadObjectType(rio *rdb);
int rdbLoad(char *filename);
int rdbSaveBackground(char *filename);
+int rdbPipesaveBackground(char *command);
+int rdbDumpsaveBackground(int fd);
void rdbRemoveTempFile(pid_t childpid);
int rdbSave(char *filename);
+int rdbSaveToFileDescriptor(FILE *file);
int rdbSaveObject(rio *rdb, robj *o);
off_t rdbSavedObjectLen(robj *o);
off_t rdbSavedObjectPages(robj *o);
View
3  src/redis.c
@@ -213,6 +213,8 @@ struct redisCommand redisCommandTable[] = {
{"ping",pingCommand,1,"r",0,NULL,0,0,0,0,0},
{"echo",echoCommand,2,"r",0,NULL,0,0,0,0,0},
{"save",saveCommand,1,"ars",0,NULL,0,0,0,0,0},
+ {"pipesave",pipesaveCommand,1,"ars",0,NULL,0,0,0,0,0},
+ {"dumpsave",dumpsaveCommand,1,"ars",0,NULL,0,0,0,0,0},
{"bgsave",bgsaveCommand,1,"ar",0,NULL,0,0,0,0,0},
{"bgrewriteaof",bgrewriteaofCommand,1,"ar",0,NULL,0,0,0,0,0},
{"shutdown",shutdownCommand,-1,"arl",0,NULL,0,0,0,0,0},
@@ -1258,6 +1260,7 @@ void initServerConfig() {
server.pidfile = zstrdup(REDIS_DEFAULT_PID_FILE);
server.rdb_filename = zstrdup(REDIS_DEFAULT_RDB_FILENAME);
server.aof_filename = zstrdup("appendonly.aof");
+ server.pipesavecommand = NULL;
server.requirepass = NULL;
server.rdb_compression = REDIS_DEFAULT_RDB_COMPRESSION;
server.rdb_checksum = REDIS_DEFAULT_RDB_CHECKSUM;
View
3  src/redis.h
@@ -824,6 +824,7 @@ struct redisServer {
time_t rdb_save_time_start; /* Current RDB save start time. */
int lastbgsave_status; /* REDIS_OK or REDIS_ERR */
int stop_writes_on_bgsave_err; /* Don't allow writes if can't BGSAVE */
+ char *pipesavecommand; /* Program to fork to receive an RDB over a pipe */
/* Propagation of commands in AOF / replication */
redisOpArray also_propagate; /* Additional command to propagate. */
/* Logging */
@@ -1387,6 +1388,8 @@ void lastsaveCommand(redisClient *c);
void saveCommand(redisClient *c);
void bgsaveCommand(redisClient *c);
void bgrewriteaofCommand(redisClient *c);
+void pipesaveCommand(redisClient *c);
+void dumpsaveCommand(redisClient *c);
void shutdownCommand(redisClient *c);
void moveCommand(redisClient *c);
void renameCommand(redisClient *c);
View
95 utils/dumpsave_client.py
@@ -0,0 +1,95 @@
+#!/usr/bin/python
+
+"""
+NAME
+
+ dumpsaveclient - example implementation of a Redis DUMPSAVE client
+
+SYNOPSIS
+
+ dumpsaveclient HOST PORT FILE
+
+DESCRIPTION
+
+ Connect to the Redis server listening on HOST:PORT and dump database
+ index 0 to FILE using the server's native compressed RDB binary
+ format.
+
+ Requires Python 2.6.
+
+EXAMPLE USAGE
+
+ % ./dumpsaveclient localhost 6379 dumpsave.rdb
+ Dumping RDB from localhost:6379 to dumpsave.rdb...
+ 18 bytes done
+
+ % src/redis-cli save
+ OK
+
+ % crc32 dump.rdb dumpsave.rdb
+ 504b1d79 dump.rdb
+ 504b1d79 dumpsave.rdb
+
+"""
+import os
+import socket
+import sys
+
+def dumpsave(redis_server, f):
+ """Issue a DUMPSAVE to `redis_server` and write the resulting RDB
+ data we get back to `f`.
+
+ `redis_server` must be a `(host, port)` 2-tuple.
+
+ `f` can be anything that behaves like a file.
+
+ """
+ BUFLEN = 4096
+
+ # http://redis.io/topics/protocol
+ QUERY_DUMPSAVE = "*1\r\n$8\r\nDUMPSAVE\r\n"
+
+ buf = bytearray(BUFLEN)
+ s = socket.create_connection(redis_server)
+
+ try:
+ s.sendall(QUERY_DUMPSAVE)
+ while True:
+ received = s.recv_into(buf, BUFLEN)
+ if received == 0:
+ break
+ f.write(buf[0:received])
+ finally:
+ try:
+ s.close()
+ except Exception:
+ pass
+
+def usage(argv):
+ print >>sys.stderr, ("Usage:\n\t%s HOST PORT FILE" %
+ os.path.basename(argv[0]))
+ return 2
+
+def main(argv=None):
+ if argv == None:
+ argv = sys.argv
+
+ try:
+ host, port, path = argv[1:4]
+ except ValueError:
+ return usage(argv)
+
+ print "Dumping RDB from %s:%s to %s..." % (host, port, path)
+
+ f = open(path, 'w')
+ try:
+ dumpsave((host, port), f)
+ finally:
+ f.close()
+
+ bytes = os.stat(path).st_size
+ print "%d bytes done" % bytes
+
+if __name__ == '__main__':
+ sys.exit(main(sys.argv))
+
Something went wrong with that request. Please try again.