Skip to content
This repository

Add a 128-bit k-ordered unique id generator #295

Closed
wants to merge 1 commit into from

6 participants

Pierre-Yves Ritschard Salvatore Sanfilippo Andrew Armstrong Tom Arnfeld Pedro Paixao Baron
Pierre-Yves Ritschard
pyr commented

This is similar to what snowflake and the recent boundary
solution do, but it makes sense to use redis for that type
of use cases for people wanting a simple way to get
incremental ids in distributed systems without an additional
daemon requirement.

Unique IDs are composed as follows:

  • epoch seconds: 4 bytes
  • epoch mseconds: 4 bytes
  • host name: 6 bytes
  • sequence id: 2 bytes

host name is truncated to 6 chars, so the appropriate config
directive id-generation-name should be set on each machines
wanting to yield ids if truncating hostname do 6 chars does
not suffice.

Pierre-Yves Ritschard Add a 128-bit k-ordered unique id generator
This is similar to what snowflake and the recent boundary
solution do, but it makes sense to use redis for that type
of use cases for people wanting a simple way to get
incremental ids in distributed systems without an additional
daemon requirement.

Unique IDs are composed as follows:

* epoch seconds: 4 bytes
* epoch mseconds: 4 bytes
* host name: 6 bytes
* sequence id: 2 bytes

host name is truncated to 6 chars, so the appropriate config
directive id-generation-name should be set on each machines
wanting to yield ids if truncating hostname do 6 chars does
not suffice.
dd86c60
Pierre-Yves Ritschard
pyr commented

A few more thoughts on this PR

  • I'm not sure about the name, seqid could be fine too
  • Here are the links to similar projects: https://github.com/twitter/snowflake, https://github.com/boundary/flake
  • This might be possible with lua too, but is only useful in scenarios where it takes huge hits and thus needs the speed of C
  • I'm not sure abut having it live in its own file, I"m willing to have it in some other file if need be
  • I wonder about the best possible way to represent a 128 bit integer from redis, and if this hex string will be sufficient
Pierre-Yves Ritschard
pyr commented

I'm wondering whether it would make sense to also provide a incrid short version which would produce 64bit snowflake like ids for people wanting shorter ids.

Salvatore Sanfilippo
Owner

What is the reason why this kind of IDs can't be generated client side? I think that exposing this things can be a bad idea, especially since here there is the notion of synchronized time between instances, a premise that all the other parts of Redis don't require.

Pierre-Yves Ritschard

The time sync is not crucial. The more in sync you are, the more sequential IDs will be. The ID behind using redis for it is simple, you can ensure sequential IDs if you have a master / slave setup with failover to the slave in case of failure, and redis would be able to handle a huge generation load.

Andrew Armstrong

Please consider adding this. An alternative is setting up another service (such as snowflake as mentioned), but that's another service that needs monitoring, testing, redundancy, etc.

Tom Arnfeld

This is exactly what I need. Do you have an idea of how small these generated numbers could start from, and how large they could get. I assume as time goes on, they'll get larger?

Pierre-Yves Ritschard
pyr commented
Salvatore Sanfilippo
Owner

we now have scripting int 2.6, and the TIME command. Is this enough to create decent IDs without adding commands?

Pedro Paixao

It could be implemented via Lua script if a bitwise library, like bitOp was loaded into redis. The important code from Snowflake is

((timestamp - twepoch) << timestampLeftShift) |
  (datacenterId << datacenterIdShift) |
  (workerId << workerIdShift) | 
  sequence
Baron

With scripting + TIME command, you could presumably generate the ID, but then have to make a separate round trip to store the ID since you can't store IDs after calling TIME in your script, correct?

Pierre-Yves Ritschard pyr closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Jan 14, 2012
Pierre-Yves Ritschard Add a 128-bit k-ordered unique id generator
This is similar to what snowflake and the recent boundary
solution do, but it makes sense to use redis for that type
of use cases for people wanting a simple way to get
incremental ids in distributed systems without an additional
daemon requirement.

Unique IDs are composed as follows:

* epoch seconds: 4 bytes
* epoch mseconds: 4 bytes
* host name: 6 bytes
* sequence id: 2 bytes

host name is truncated to 6 chars, so the appropriate config
directive id-generation-name should be set on each machines
wanting to yield ids if truncating hostname do 6 chars does
not suffice.
dd86c60
This page is out of date. Refresh to see the latest.
4  src/Makefile
@@ -73,7 +73,7 @@ QUIET_CC = @printf '    %b %b\n' $(CCCOLOR)CC$(ENDCOLOR) $(SRCCOLOR)$@$(ENDCOLOR
73 73
 QUIET_LINK = @printf '    %b %b\n' $(LINKCOLOR)LINK$(ENDCOLOR) $(BINCOLOR)$@$(ENDCOLOR);
74 74
 endif
75 75
 
76  
-OBJ = adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endian.o slowlog.o scripting.o bio.o rio.o rand.o
  76
+OBJ = adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o id.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endian.o slowlog.o scripting.o bio.o rio.o rand.o
77 77
 BENCHOBJ = ae.o anet.o redis-benchmark.o sds.o adlist.o zmalloc.o
78 78
 CLIOBJ = anet.o sds.o adlist.o redis-cli.o zmalloc.o release.o
79 79
 CHECKDUMPOBJ = redis-check-dump.o lzf_c.o lzf_d.o
@@ -107,6 +107,8 @@ config.o: config.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
107 107
   zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
108 108
 crc16.o: crc16.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
109 109
   zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
  110
+id.o: id.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
  111
+  zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
110 112
 db.o: db.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
111 113
   zmalloc.h anet.h zipmap.h ziplist.h intset.h version.h util.h
112 114
 debug.o: debug.c redis.h fmacros.h config.h ae.h sds.h dict.h adlist.h \
15  src/config.c
@@ -305,6 +305,11 @@ void loadServerConfigFromString(char *config) {
305 305
             server.cluster.configfile = zstrdup(argv[1]);
306 306
         } else if (!strcasecmp(argv[0],"lua-time-limit") && argc == 2) {
307 307
             server.lua_time_limit = strtoll(argv[1],NULL,10);
  308
+	} else if (!strcasecmp(argv[0], "id-generation-name") && argc == 2) {
  309
+            int len = (strlen(argv[1]) < REDIS_ID_NAMELEN) ?
  310
+                strlen(argv[1]) : REDIS_ID_NAMELEN;
  311
+            bzero(server.id_generation_name, REDIS_ID_NAMESPACE);
  312
+	    memcpy(server.id_generation_name, argv[1], len);
308 313
         } else if (!strcasecmp(argv[0],"slowlog-log-slower-than") &&
309 314
                    argc == 2)
310 315
         {
@@ -515,6 +520,11 @@ void configSetCommand(redisClient *c) {
515 520
     } else if (!strcasecmp(c->argv[2]->ptr,"lua-time-limit")) {
516 521
         if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt;
517 522
         server.lua_time_limit = ll;
  523
+    } else if (!strcasecmp(c->argv[1]->ptr,"id-generation-name")) {
  524
+        int len = (strlen((char *)o->ptr) < REDIS_ID_NAMELEN) ?
  525
+            strlen((char *)o->ptr) : REDIS_ID_NAMELEN;
  526
+        bzero(server.id_generation_name, REDIS_ID_NAMESPACE);
  527
+        memcpy(server.id_generation_name, o->ptr, len);
518 528
     } else if (!strcasecmp(c->argv[2]->ptr,"slowlog-log-slower-than")) {
519 529
         if (getLongLongFromObject(o,&ll) == REDIS_ERR) goto badfmt;
520 530
         server.slowlog_log_slower_than = ll;
@@ -708,6 +718,11 @@ void configGetCommand(redisClient *c) {
708 718
         addReplyBulkLongLong(c,server.lua_time_limit);
709 719
         matches++;
710 720
     }
  721
+    if (stringmatch(pattern,"id-generation-name",0)) {
  722
+        addReplyBulkCString(c,"id-generation-name");
  723
+        addReplyBulkCString(c,server.id_generation_name);
  724
+        matches++;
  725
+    }
711 726
     if (stringmatch(pattern,"slowlog-log-slower-than",0)) {
712 727
         addReplyBulkCString(c,"slowlog-log-slower-than");
713 728
         addReplyBulkLongLong(c,server.slowlog_log_slower_than);
27  src/id.c
... ...
@@ -0,0 +1,27 @@
  1
+#include "redis.h"
  2
+#include <sys/time.h>
  3
+
  4
+u_int16_t sequence;
  5
+
  6
+void incridCommand(redisClient *c) {
  7
+    struct timeval tv;
  8
+    sds id_buf;
  9
+
  10
+    if ((gettimeofday(&tv, NULL)) == -1) {
  11
+        addReplyError(c, "cannot get time of day");
  12
+	return;
  13
+    }
  14
+
  15
+    id_buf = sdscatprintf(sdsempty(),
  16
+                          "0x%08lx%08x%02x%02x%02x%02x%02x%02x%04x\r\n",
  17
+			  tv.tv_sec,
  18
+                          tv.tv_usec,
  19
+                          server.id_generation_name[0],
  20
+                          server.id_generation_name[1],
  21
+                          server.id_generation_name[2],
  22
+                          server.id_generation_name[3],
  23
+                          server.id_generation_name[4],
  24
+                          server.id_generation_name[5],
  25
+                          sequence++);
  26
+    addReplySds(c, id_buf);
  27
+}
8  src/redis.c
@@ -53,6 +53,7 @@
53 53
 #include <float.h>
54 54
 #include <math.h>
55 55
 #include <sys/resource.h>
  56
+#include <string.h>
56 57
 
57 58
 /* Our shared "common" objects */
58 59
 
@@ -244,7 +245,8 @@ struct redisCommand redisCommandTable[] = {
244 245
     {"eval",evalCommand,-3,"wms",0,zunionInterGetKeys,0,0,0,0,0},
245 246
     {"evalsha",evalShaCommand,-3,"wms",0,zunionInterGetKeys,0,0,0,0,0},
246 247
     {"slowlog",slowlogCommand,-2,"r",0,NULL,0,0,0,0,0},
247  
-    {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0}
  248
+    {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0},
  249
+    {"incrid",incridCommand,1,"rR",0,NULL,0,0,0,0,0}
248 250
 };
249 251
 
250 252
 /*============================ Utility functions ============================ */
@@ -859,6 +861,7 @@ void createSharedObjects(void) {
859 861
 }
860 862
 
861 863
 void initServerConfig() {
  864
+    
862 865
     server.port = REDIS_SERVERPORT;
863 866
     server.bindaddr = NULL;
864 867
     server.unixsocket = NULL;
@@ -949,6 +952,9 @@ void initServerConfig() {
949 952
     server.slowlog_log_slower_than = REDIS_SLOWLOG_LOG_SLOWER_THAN;
950 953
     server.slowlog_max_len = REDIS_SLOWLOG_MAX_LEN;
951 954
 
  955
+    /* Hostname */
  956
+    (void)gethostname(server.id_generation_name, REDIS_ID_NAMESPACE);
  957
+
952 958
     /* Assert */
953 959
     server.assert_failed = "<no assertion failed>";
954 960
     server.assert_file = "<no file>";
7  src/redis.h
@@ -222,6 +222,10 @@
222 222
 #define UNIT_SECONDS 0
223 223
 #define UNIT_MILLISECONDS 1
224 224
 
  225
+/* ID Generation */
  226
+#define REDIS_ID_NAMELEN 6
  227
+#define REDIS_ID_NAMESPACE 256
  228
+
225 229
 /* SHUTDOWN flags */
226 230
 #define REDIS_SHUTDOWN_SAVE 1       /* Force SAVE on SHUTDOWN even if no save
227 231
                                        points are configured. */
@@ -644,6 +648,8 @@ struct redisServer {
644 648
     char *assert_file;
645 649
     int assert_line;
646 650
     int bug_report_start; /* True if bug report header was already logged. */
  651
+    char id_generation_name[REDIS_ID_NAMESPACE];
  652
+         /* hostname to include in generated sequence ids. */
647 653
 };
648 654
 
649 655
 typedef struct pubsubPattern {
@@ -1147,6 +1153,7 @@ void clientCommand(redisClient *c);
1147 1153
 void evalCommand(redisClient *c);
1148 1154
 void evalShaCommand(redisClient *c);
1149 1155
 void scriptCommand(redisClient *c);
  1156
+void incridCommand(redisClient *c);
1150 1157
 
1151 1158
 #if defined(__GNUC__)
1152 1159
 void *calloc(size_t count, size_t size) __attribute__ ((deprecated));
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.