Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Add a 128-bit k-ordered unique id generator #295

Closed
wants to merge 1 commit into from

6 participants

@pyr

This is similar to what snowflake and the recent boundary
solution do, but it makes sense to use redis for that type
of use cases for people wanting a simple way to get
incremental ids in distributed systems without an additional
daemon requirement.

Unique IDs are composed as follows:

  • epoch seconds: 4 bytes
  • epoch mseconds: 4 bytes
  • host name: 6 bytes
  • sequence id: 2 bytes

host name is truncated to 6 chars, so the appropriate config
directive id-generation-name should be set on each machines
wanting to yield ids if truncating hostname do 6 chars does
not suffice.

@pyr pyr Add a 128-bit k-ordered unique id generator
This is similar to what snowflake and the recent boundary
solution do, but it makes sense to use redis for that type
of use cases for people wanting a simple way to get
incremental ids in distributed systems without an additional
daemon requirement.

Unique IDs are composed as follows:

* epoch seconds: 4 bytes
* epoch mseconds: 4 bytes
* host name: 6 bytes
* sequence id: 2 bytes

host name is truncated to 6 chars, so the appropriate config
directive id-generation-name should be set on each machines
wanting to yield ids if truncating hostname do 6 chars does
not suffice.
dd86c60
@pyr

A few more thoughts on this PR

  • I'm not sure about the name, seqid could be fine too
  • Here are the links to similar projects: https://github.com/twitter/snowflake, https://github.com/boundary/flake
  • This might be possible with lua too, but is only useful in scenarios where it takes huge hits and thus needs the speed of C
  • I'm not sure abut having it live in its own file, I"m willing to have it in some other file if need be
  • I wonder about the best possible way to represent a 128 bit integer from redis, and if this hex string will be sufficient
@pyr

I'm wondering whether it would make sense to also provide a incrid short version which would produce 64bit snowflake like ids for people wanting shorter ids.

@antirez
Owner

What is the reason why this kind of IDs can't be generated client side? I think that exposing this things can be a bad idea, especially since here there is the notion of synchronized time between instances, a premise that all the other parts of Redis don't require.

@pyr
pyr commented

The time sync is not crucial. The more in sync you are, the more sequential IDs will be. The ID behind using redis for it is simple, you can ensure sequential IDs if you have a master / slave setup with failover to the slave in case of failure, and redis would be able to handle a huge generation load.

@Plasma

Please consider adding this. An alternative is setting up another service (such as snowflake as mentioned), but that's another service that needs monitoring, testing, redundancy, etc.

@tarnfeld

This is exactly what I need. Do you have an idea of how small these generated numbers could start from, and how large they could get. I assume as time goes on, they'll get larger?

@pyr
@antirez
Owner

we now have scripting int 2.6, and the TIME command. Is this enough to create decent IDs without adding commands?

@paixaop

It could be implemented via Lua script if a bitwise library, like bitOp was loaded into redis. The important code from Snowflake is

((timestamp - twepoch) << timestampLeftShift) |
  (datacenterId << datacenterIdShift) |
  (workerId << workerIdShift) | 
  sequence
@breznik

With scripting + TIME command, you could presumably generate the ID, but then have to make a separate round trip to store the ID since you can't store IDs after calling TIME in your script, correct?

@pyr pyr closed this
@JackieXie168 JackieXie168 referenced this pull request from a commit
@timmaxw timmaxw Made slave resume independent behavior on startup if it was independe…
…nt of master at the time it shut down. Closes #295.
d70075f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 13, 2012
  1. @pyr

    Add a 128-bit k-ordered unique id generator

    pyr authored
    This is similar to what snowflake and the recent boundary
    solution do, but it makes sense to use redis for that type
    of use cases for people wanting a simple way to get
    incremental ids in distributed systems without an additional
    daemon requirement.
    
    Unique IDs are composed as follows:
    
    * epoch seconds: 4 bytes
    * epoch mseconds: 4 bytes
    * host name: 6 bytes
    * sequence id: 2 bytes
    
    host name is truncated to 6 chars, so the appropriate config
    directive id-generation-name should be set on each machines
    wanting to yield ids if truncating hostname do 6 chars does
    not suffice.
This page is out of date. Refresh to see the latest.
View
4 src/Makefile
@@ -73,7 +73,7 @@ QUIET_CC = @printf ' %b %b\n' $(CCCOLOR)CC$(ENDCOLOR) $(SRCCOLOR)$@$(ENDCOLOR
QUIET_LINK = @printf ' %b %b\n' $(LINKCOLOR)LINK$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR);
endif
-OBJ = adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endian.o slowlog.o scripting.o bio.o rio.o rand.o
+OBJ = adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o id.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endian.o slowlog.o scripting.o bio.o rio.o rand.o
BENCHOBJ = ae.o anet.o redis-benchmark.o sds.o adlist.o zmalloc.o
CLIOBJ = anet.o sds.o adlist.o redis-cli.o zmalloc.o release.o
CHECKDUMPOBJ = redis-check-dump.o lzf_c.o lzf_d.o
@@ -107,6 +107,8 @@ config.o: config.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
crc16.o: crc16.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
+id.o: id.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
+ zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
db.o: db.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
debug.o: debug.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
View
15 src/config.c
@@ -305,6 +305,11 @@ void loadServerConfigFromString(char *config) {
server.cluster.configfile = zstrdup(argv[1]);
} else if (!strcasecmp(argv[0],"lua-time-limit") && argc == 2) {
server.lua_time_limit = strtoll(argv[1],NULL,10);
+ } else if (!strcasecmp(argv[0], "id-generation-name") && argc == 2) {
+ int len = (strlen(argv[1]) < REDIS_ID_NAMELEN) ?
+ strlen(argv[1]) : REDIS_ID_NAMELEN;
+ bzero(server.id_generation_name, REDIS_ID_NAMESPACE);
+ memcpy(server.id_generation_name, argv[1], len);
} else if (!strcasecmp(argv[0],"slowlog-log-slower-than") &&
argc == 2)
{
@@ -515,6 +520,11 @@ void configSetCommand(redisClient *c) {
} else if (!strcasecmp(c->argv[2]->ptr,"lua-time-limit")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt;
server.lua_time_limit = ll;
+ } else if (!strcasecmp(c->argv[1]->ptr,"id-generation-name")) {
+ int len = (strlen((char *)o->ptr) < REDIS_ID_NAMELEN) ?
+ strlen((char *)o->ptr) : REDIS_ID_NAMELEN;
+ bzero(server.id_generation_name, REDIS_ID_NAMESPACE);
+ memcpy(server.id_generation_name, o->ptr, len);
} else if (!strcasecmp(c->argv[2]->ptr,"slowlog-log-slower-than")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR) goto badfmt;
server.slowlog_log_slower_than = ll;
@@ -708,6 +718,11 @@ void configGetCommand(redisClient *c) {
addReplyBulkLongLong(c,server.lua_time_limit);
matches++;
}
+ if (stringmatch(pattern,"id-generation-name",0)) {
+ addReplyBulkCString(c,"id-generation-name");
+ addReplyBulkCString(c,server.id_generation_name);
+ matches++;
+ }
if (stringmatch(pattern,"slowlog-log-slower-than",0)) {
addReplyBulkCString(c,"slowlog-log-slower-than");
addReplyBulkLongLong(c,server.slowlog_log_slower_than);
View
27 src/id.c
@@ -0,0 +1,27 @@
+#include "redis.h"
+#include <sys/time.h>
+
+u_int16_t sequence;
+
+void incridCommand(redisClient *c) {
+ struct timeval tv;
+ sds id_buf;
+
+ if ((gettimeofday(&tv, NULL)) == -1) {
+ addReplyError(c, "cannot get time of day");
+ return;
+ }
+
+ id_buf = sdscatprintf(sdsempty(),
+ "0x%08lx%08x%02x%02x%02x%02x%02x%02x%04x\r\n",
+ tv.tv_sec,
+ tv.tv_usec,
+ server.id_generation_name[0],
+ server.id_generation_name[1],
+ server.id_generation_name[2],
+ server.id_generation_name[3],
+ server.id_generation_name[4],
+ server.id_generation_name[5],
+ sequence++);
+ addReplySds(c, id_buf);
+}
View
8 src/redis.c
@@ -53,6 +53,7 @@
#include <float.h>
#include <math.h>
#include <sys/resource.h>
+#include <string.h>
/* Our shared "common" objects */
@@ -244,7 +245,8 @@ struct redisCommand redisCommandTable[] = {
{"eval",evalCommand,-3,"wms",0,zunionInterGetKeys,0,0,0,0,0},
{"evalsha",evalShaCommand,-3,"wms",0,zunionInterGetKeys,0,0,0,0,0},
{"slowlog",slowlogCommand,-2,"r",0,NULL,0,0,0,0,0},
- {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0}
+ {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0},
+ {"incrid",incridCommand,1,"rR",0,NULL,0,0,0,0,0}
};
/*============================ Utility functions ============================ */
@@ -859,6 +861,7 @@ void createSharedObjects(void) {
}
void initServerConfig() {
+
server.port = REDIS_SERVERPORT;
server.bindaddr = NULL;
server.unixsocket = NULL;
@@ -949,6 +952,9 @@ void initServerConfig() {
server.slowlog_log_slower_than = REDIS_SLOWLOG_LOG_SLOWER_THAN;
server.slowlog_max_len = REDIS_SLOWLOG_MAX_LEN;
+ /* Hostname */
+ (void)gethostname(server.id_generation_name, REDIS_ID_NAMESPACE);
+
/* Assert */
server.assert_failed = "<no assertion failed>";
server.assert_file = "<no file>";
View
7 src/redis.h
@@ -222,6 +222,10 @@
#define UNIT_SECONDS 0
#define UNIT_MILLISECONDS 1
+/* ID Generation */
+#define REDIS_ID_NAMELEN 6
+#define REDIS_ID_NAMESPACE 256
+
/* SHUTDOWN flags */
#define REDIS_SHUTDOWN_SAVE 1 /* Force SAVE on SHUTDOWN even if no save
points are configured. */
@@ -644,6 +648,8 @@ struct redisServer {
char *assert_file;
int assert_line;
int bug_report_start; /* True if bug report header was already logged. */
+ char id_generation_name[REDIS_ID_NAMESPACE];
+ /* hostname to include in generated sequence ids. */
};
typedef struct pubsubPattern {
@@ -1147,6 +1153,7 @@ void clientCommand(redisClient *c);
void evalCommand(redisClient *c);
void evalShaCommand(redisClient *c);
void scriptCommand(redisClient *c);
+void incridCommand(redisClient *c);
#if defined(__GNUC__)
void *calloc(size_t count, size_t size) __attribute__ ((deprecated));
Something went wrong with that request. Please try again.