Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Added lazy_connect option to RedisArray #303

Merged
merged 17 commits into from

2 participants

@mobli

Hi Michael,

So here we go, my first try after using git flow.

Here is the summary of the changes I'm proposing:
Added an option to let each RedisArray connection connect lazily to their respective server. This is useful then working with a redis cluster composed of many shards which are not necessarily in use all at once.

Cheers!

Emmanuel

mobli added some commits
@mobli mobli Retry delay - selectDB on array
Added the possibility to delay each reconnection attempt, including a
random factor to prevent several or many concurrent connections from
trying to reconnect at the same time.
Added the select command to RedisArray to select a DB on every
connections in one instruction.
3fb6432
@mobli mobli New select DB command to RedisArray - Added retry delay on reconnect
Added the possibility to delay each reconnection attempt, including a
random factor to prevent several or many concurrent connections from
trying to reconnect at the same time.
Added the select command to RedisArray to select a DB on every
connections in one instruction.
Also, fixed a compiler warning:
redis_array_impl.c:1115:15: warning: incompatible pointer types
assigning to 'zval **' (aka 'struct _zval_struct **') from 'zval
**(*)[2]' [-Wincompatible-pointer-types]
6091727
@mobli mobli Revert "New select DB command to RedisArray - Added retry delay on re…
…connect"

This reverts commit 6091727.
e0ee405
@mobli mobli Revert "Retry delay - selectDB on array"
This reverts commit 3fb6432.
9ba000c
@mobli mobli New select DB command to RedisArray - Added retry delay on reconnect
Added the possibility to delay each reconnection attempt, including a
random factor to prevent several or many concurrent connections from
trying to reconnect at the same time.
Added the select command to RedisArray to select a DB on every
connections in one instruction.
Also, fixed a compiler warning:
redis_array_impl.c:1115:15: warning: incompatible pointer types
assigning to 'zval **' (aka 'struct _zval_struct **') from 'zval
**(*)[2]' [-Wincompatible-pointer-types]
d4c7f64
@mobli mobli Merge remote-tracking branch 'upstream/master'
Conflicts:
	common.h
4ae635f
@mobli mobli Added lazy_connect option to RedisArray
Added an option to let each RedisArray connection connect lazily to
their respective server. This is useful then working with a redis
cluster composed of many shards which are not necessarily in use all at
once.
acd3245
@mobli mobli Merge remote-tracking branch 'upstream/develop' into develop
Conflicts:
	library.c
	library.h
	redis.c
	redis_array.c
	redis_array_impl.c
	redis_array_impl.h
	redis_session.c
ec74ab9
@mobli mobli Added lazy_connect option to RedisArray
Added an option to let each RedisArray connection connect lazily to
their respective server. This is useful then working with a redis
cluster composed of many shards which are not necessarily in use all at
once.
1f3358b
@mobli

Also, the extension doesn't compile in the develop branch. I've compiled locally by making some changes to library.c, but I didn't want to commit them because I'm not sure what the intention is. For instance:

library.c:707:21: error: void function 'redis_client_list_reply' should not return a value

I replaced

return -1

with

RETURN_FALSE

Also in

library.c:852:13: error: non-void function 'redis_sock_read_multibulk_reply_zipped_with_flag' should return a value

I simply removed the lines

        } else {
            RETURN_FALSE;

Same for the remaining errors.
So, it compiles but I'm not sure what it breaks...

@michael-grunder

Nice spot! Those are things that I broke :) I'll get them sorted for sure.

I think we're using Redis in a similar way (tons of shards, all over the place). I thing an option to connect lazily is a good thing too.

Cheers!
Mike

@mobli

Hi Michael (can I call you Mike? :-))
My concern is that the code that breaks compilation is also in master. If I want to build a clean "official" redis.so, I can't just clone the repo now. What do I need to do?

Thanks,

Emmanuel

@michael-grunder

Hey there,

Yes, Michael, Mike, or "hey you" are all acceptable. :)

I have merged the client reply fix into develop and hotfixed the zipped_with_reply issue.

Cheers dude,
Mike

@mobli

Hi again,

There are still 3 compilation errors of the same kind (non-void function 'bla-bla' should return a value) on lines 1096, 1143 and 1222. I don't really get the point of these, why not simply returning -1?

Cheers,

Emmanuel

@michael-grunder michael-grunder referenced this pull request from a commit
@michael-grunder michael-grunder Fix each place we're erroring out but return a value as the
methods are int returns

Resolves issues pertaining to #303
703476c
@michael-grunder

Hey dude, the reason that it's not simply returning -1 is that we're setting the return value for the PHP userland caller. If we're not in a pipeline we want PHP to return FALSE. if we are in a pipeline, we want to add FALSE to the array of pipeline/multi replies.

This looks to get all of them.

Cheers,
Mike

@mobli

Aha! I get it! Thanks for the explanation, now it all makes sense!

Have a good night now (if you ever sleep) and thanks a lot for the really fast turnaround!

Take care,

Emmanuel

mobli added some commits
@mobli mobli Merge remote-tracking branch 'upstream/develop' into develop 4f2aa60
@mobli mobli Easy resharding distributor
An integrated distributor for easy resharding.
Works on the principle of redistributing keys from 1 shard to 2 shards
evenly.
f4acf12
@mobli mobli Easy resharding distributor
Updated documentation
6b30539
@mobli mobli Easy resharding distributor
Updated documentation
d71ed81
@mobli mobli Easy resharding distributor
Updated documentation
01f0930
@mobli mobli FIx C99 compliance
FIx C99 compliance
2f795ed
@mobli mobli Changed distributor to accepts strings
Changed distributor to accept strings as well as longs so that
definitions parsed from parse_ini_files may be used as is
958e062
@mobli mobli Fix poor performance on initial use
Fix poor performance on initial use of easy reshard distributor
mechanism.
0ff393b
@michael-grunder

Hey there,

Sorry for such a long delay. I have merged and integrated your changes into the feature branch feature/mobli_ra_changes along with fixing some conflicts. I would like to do some testing before getting this into develop and then into master, but I figured I would let you know.

Cheers!
Mike

@michael-grunder michael-grunder merged commit 0ff393b into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 21, 2013
  1. @mobli

    Retry delay - selectDB on array

    mobli authored
    Added the possibility to delay each reconnection attempt, including a
    random factor to prevent several or many concurrent connections from
    trying to reconnect at the same time.
    Added the select command to RedisArray to select a DB on every
    connections in one instruction.
Commits on Jan 22, 2013
  1. @mobli

    New select DB command to RedisArray - Added retry delay on reconnect

    mobli authored
    Added the possibility to delay each reconnection attempt, including a
    random factor to prevent several or many concurrent connections from
    trying to reconnect at the same time.
    Added the select command to RedisArray to select a DB on every
    connections in one instruction.
    Also, fixed a compiler warning:
    redis_array_impl.c:1115:15: warning: incompatible pointer types
    assigning to 'zval **' (aka 'struct _zval_struct **') from 'zval
    **(*)[2]' [-Wincompatible-pointer-types]
Commits on Jan 29, 2013
  1. @mobli

    Revert "New select DB command to RedisArray - Added retry delay on re…

    mobli authored
    …connect"
    
    This reverts commit 6091727.
  2. @mobli

    Revert "Retry delay - selectDB on array"

    mobli authored
    This reverts commit 3fb6432.
  3. @mobli

    New select DB command to RedisArray - Added retry delay on reconnect

    mobli authored
    Added the possibility to delay each reconnection attempt, including a
    random factor to prevent several or many concurrent connections from
    trying to reconnect at the same time.
    Added the select command to RedisArray to select a DB on every
    connections in one instruction.
    Also, fixed a compiler warning:
    redis_array_impl.c:1115:15: warning: incompatible pointer types
    assigning to 'zval **' (aka 'struct _zval_struct **') from 'zval
    **(*)[2]' [-Wincompatible-pointer-types]
Commits on Feb 24, 2013
  1. @mobli

    Merge remote-tracking branch 'upstream/master'

    mobli authored
    Conflicts:
    	common.h
  2. @mobli

    Added lazy_connect option to RedisArray

    mobli authored
    Added an option to let each RedisArray connection connect lazily to
    their respective server. This is useful then working with a redis
    cluster composed of many shards which are not necessarily in use all at
    once.
  3. @mobli

    Merge remote-tracking branch 'upstream/develop' into develop

    mobli authored
    Conflicts:
    	library.c
    	library.h
    	redis.c
    	redis_array.c
    	redis_array_impl.c
    	redis_array_impl.h
    	redis_session.c
  4. @mobli

    Added lazy_connect option to RedisArray

    mobli authored
    Added an option to let each RedisArray connection connect lazily to
    their respective server. This is useful then working with a redis
    cluster composed of many shards which are not necessarily in use all at
    once.
Commits on Feb 25, 2013
  1. @mobli
Commits on May 22, 2013
  1. @mobli

    Easy resharding distributor

    mobli authored
    An integrated distributor for easy resharding.
    Works on the principle of redistributing keys from 1 shard to 2 shards
    evenly.
  2. @mobli

    Easy resharding distributor

    mobli authored
    Updated documentation
  3. @mobli

    Easy resharding distributor

    mobli authored
    Updated documentation
  4. @mobli

    Easy resharding distributor

    mobli authored
    Updated documentation
Commits on May 25, 2013
  1. @mobli

    FIx C99 compliance

    mobli authored
    FIx C99 compliance
Commits on Jul 22, 2013
  1. @mobli

    Changed distributor to accepts strings

    mobli authored
    Changed distributor to accept strings as well as longs so that
    definitions parsed from parse_ini_files may be used as is
  2. @mobli

    Fix poor performance on initial use

    mobli authored
    Fix poor performance on initial use of easy reshard distributor
    mechanism.
This page is out of date. Refresh to see the latest.
View
4 README.markdown
@@ -164,6 +164,8 @@ _**Description**_: Connects to a Redis instance.
*host*: string. can be a host, or the path to a unix domain socket
*port*: int, optional
*timeout*: float, value in seconds (optional, default is 0 meaning unlimited)
+*reserved*: should be NULL if retry_interval is specified
+*retry_interval*: int, value in milliseconds (optional)
##### *Return value*
@@ -176,6 +178,7 @@ $redis->connect('127.0.0.1', 6379);
$redis->connect('127.0.0.1'); // port 6379 by default
$redis->connect('127.0.0.1', 6379, 2.5); // 2.5 sec timeout.
$redis->connect('/tmp/redis.sock'); // unix domain socket.
+$redis->connect('127.0.0.1', 6379, 1, NULL, 100); // 1 sec timeout, 100ms delay between reconnection attempts.
~~~
### pconnect, popen
@@ -198,6 +201,7 @@ persistent equivalents.
*port*: int, optional
*timeout*: float, value in seconds (optional, default is 0 meaning unlimited)
*persistent_id*: string. identity for the requested persistent connection
+*retry_interval*: int, value in milliseconds (optional)
##### *Return value*
View
65 arrays.markdown
@@ -17,7 +17,7 @@ There are several ways of creating Redis arrays; they can be pre-defined in red
#### Declaring a new array with a list of nodes
<pre>
-$ra = new RedisArray(array("host1", "host2:63792, "host2:6380"));
+$ra = new RedisArray(array("host1", "host2:63792", "host2:6380"));
</pre>
@@ -26,7 +26,7 @@ $ra = new RedisArray(array("host1", "host2:63792, "host2:6380"));
function extract_key_part($k) {
return substr($k, 0, 3); // hash only on first 3 characters.
}
-$ra = new RedisArray(array("host1", "host2:63792, "host2:6380"), array("function" => "extract_key_part"));
+$ra = new RedisArray(array("host1", "host2:63792", "host2:6380"), array("function" => "extract_key_part"));
</pre>
#### Defining a "previous" array when nodes are added or removed.
@@ -34,7 +34,19 @@ When a new node is added to an array, phpredis needs to know about it. The old l
<pre>
// adding host3 to a ring containing host1 and host2. Read commands will look in the previous ring if the data is not found in the main ring.
-$ra = new RedisArray(array('host1', 'host2', 'host3'), array('previous' => array('host1', 'host2')));
+$ra = new RedisArray(array("host1", "host2", "host3"), array("previous" => array("host1", "host2")));
+</pre>
+
+#### Specifying the "retry_interval" parameter
+The retry_interval is used to specify a delay in milliseconds between reconnection attempts in case the client loses connection with a server
+<pre>
+$ra = new RedisArray(array("host1", "host2:63792", "host2:6380"), array("retry_timeout" => 100)));
+</pre>
+
+#### Specifying the "lazy_connect" parameter
+This option is useful when a cluster has many shards but not of them are necessarily used at one time.
+<pre>
+$ra = new RedisArray(array("host1", "host2:63792", "host2:6380"), array("lazy_connect" => true)));
</pre>
#### Defining arrays in Redis.ini
@@ -76,6 +88,53 @@ In order to control the distribution of keys by hand, you can provide a custom f
For instance, instanciate a RedisArray object with `new RedisArray(array("us-host", "uk-host", "de-host"), array("distributor" => "dist"));` and write a function called "dist" that will return `2` for all the keys that should end up on the "de-host" server.
+You may also provide an array of 2 values that will be used as follows:
+- The first value is the initial amount of shards in use before the resharding (the x first shards specified in the constructor)
+- The second value is the resharding level, or number of resharding iterations.
+
+For instance, suppose you started with 4 shards as follows:
+<pre>
+0 => 0 1 2 3
+</pre>
+
+After 1 iteration of resharding, keys will be assigned to the following servers:
+<pre>
+1 => 0 4 1 5 2 6 3 7
+</pre>
+
+After 2 iterations, keys will be assigned to the following servers:
+<pre>
+2 => 0 8 4 12 1 9 5 13 2 10 6 14 3 11 7 15
+</pre>
+
+After 3 iterations, keys will be assigned to the following servers:
+<pre>
+3 => 0 16 8 24 4 20 12 28 1 17 9 25 5 21 13 29 2 18 10 26 6 22 14 30 3 19 11 27 7 23 15 31
+</pre>
+
+And so on...
+
+The idea here is to be able to reshard the keys easily, without moving keys from 1 server to another.
+
+The procedure to adopt is simple:
+
+For each initial shard, setup a slave. For instance, for shard 1 we setup slave 5.
+
+Keys will now be assigned to either shard 1 or shard 5. Once the application sees the new settings, just setup shard 5 as a master. Then, in order to reclaim memory, just cleanup keys from shard 1 that belong to shard 5 and vice-versa.
+
+On the next iteration, setup a new slave 9 for shard 1 and a new slave 13 for shard 5.
+
+Update the application settings, disconnect the new slaves and clean up the shards from keys that don't belong there anymore.
+
+Apply the same procedure for each resharding iteration.
+
+### Example
+<pre>
+$ra = new RedisArray(array("host1", "host2", "host3", "host4", "host5", "host6", "host7", "host8"), array("distributor" => array(2, 2)));
+</pre>
+
+This declares that we started with 2 shards and moved to 4 then 8 shards. The number of initial shards is 2 and the resharding level (or number of iterations) is 2.
+
## Migrating keys
When a node is added or removed from a ring, RedisArray instances must be instanciated with a “previous” list of nodes. A single call to `$ra->_rehash()` causes all the keys to be redistributed according to the new list of nodes. Passing a callback function to `_rehash()` makes it possible to track the progress of that operation: the function is called with a node name and a number of keys that will be examined, e.g. `_rehash(function ($host, $count){ ... });`.
View
1  common.h
@@ -181,6 +181,7 @@ typedef struct {
char *err;
int err_len;
+ zend_bool lazy_connect;
} RedisSock;
/* }}} */
View
4 library.c
@@ -970,7 +970,8 @@ PHPAPI void redis_ping_response(INTERNAL_FUNCTION_PARAMETERS, RedisSock *redis_s
*/
PHPAPI RedisSock* redis_sock_create(char *host, int host_len, unsigned short port,
double timeout, int persistent, char *persistent_id,
- long retry_interval)
+ long retry_interval,
+ zend_bool lazy_connect)
{
RedisSock *redis_sock;
@@ -982,6 +983,7 @@ PHPAPI RedisSock* redis_sock_create(char *host, int host_len, unsigned short por
redis_sock->dbNumber = 0;
redis_sock->retry_interval = retry_interval * 1000;
redis_sock->persistent = persistent;
+ redis_sock->lazy_connect = lazy_connect;
if(persistent_id) {
size_t persistent_id_len = strlen(persistent_id);
View
2  library.h
@@ -20,7 +20,7 @@ PHPAPI void redis_string_response(INTERNAL_FUNCTION_PARAMETERS, RedisSock *redis
PHPAPI void redis_ping_response(INTERNAL_FUNCTION_PARAMETERS, RedisSock *redis_sock, zval *z_tab, void *ctx);
PHPAPI void redis_info_response(INTERNAL_FUNCTION_PARAMETERS, RedisSock *redis_sock, zval *z_tab, void *ctx);
PHPAPI void redis_type_response(INTERNAL_FUNCTION_PARAMETERS, RedisSock *redis_sock, zval *z_tab, void *ctx);
-PHPAPI RedisSock* redis_sock_create(char *host, int host_len, unsigned short port, double timeout, int persistent, char *persistent_id, long retry_interval);
+PHPAPI RedisSock* redis_sock_create(char *host, int host_len, unsigned short port, double timeout, int persistent, char *persistent_id, long retry_interval, zend_bool lazy_connect);
PHPAPI int redis_sock_connect(RedisSock *redis_sock TSRMLS_DC);
PHPAPI int redis_sock_server_open(RedisSock *redis_sock, int force_connect TSRMLS_DC);
PHPAPI int redis_sock_disconnect(RedisSock *redis_sock TSRMLS_DC);
View
9 redis.c
@@ -378,6 +378,13 @@ PHPAPI int redis_sock_get(zval *id, RedisSock **redis_sock TSRMLS_DC, int no_thr
}
return -1;
}
+ if ((*redis_sock)->lazy_connect)
+ {
+ (*redis_sock)->lazy_connect = 0;
+ if (redis_sock_server_open(*redis_sock, 1 TSRMLS_CC) < 0) {
+ return -1;
+ }
+ }
return Z_LVAL_PP(socket);
}
@@ -604,7 +611,7 @@ PHPAPI int redis_connect(INTERNAL_FUNCTION_PARAMETERS, int persistent) {
zend_clear_exception(TSRMLS_C); /* clear exception triggered by non-existent socket during connect(). */
}
- redis_sock = redis_sock_create(host, host_len, port, timeout, persistent, persistent_id, retry_interval);
+ redis_sock = redis_sock_create(host, host_len, port, timeout, persistent, persistent_id, retry_interval, 0);
if (redis_sock_server_open(redis_sock, 1 TSRMLS_CC) < 0) {
redis_free_socket(redis_sock);
View
10 redis_array.c
@@ -194,6 +194,7 @@ PHP_METHOD(RedisArray, __construct)
zend_bool b_index = 0, b_autorehash = 0;
HashTable *hPrev = NULL, *hOpts = NULL;
long l_retry_interval = 0;
+ zend_bool b_lazy_connect = 0;
if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "z|a", &z0, &z_opts) == FAILURE) {
RETURN_FALSE;
@@ -236,7 +237,7 @@ PHP_METHOD(RedisArray, __construct)
}
/* extract retry_interval option. */
- zval **z_retry_interval_pp;
+ zval **z_retry_interval_pp;
if (FAILURE != zend_hash_find(hOpts, "retry_interval", sizeof("retry_interval"), (void**)&z_retry_interval_pp)) {
if (Z_TYPE_PP(z_retry_interval_pp) == IS_LONG || Z_TYPE_PP(z_retry_interval_pp) == IS_STRING) {
if (Z_TYPE_PP(z_retry_interval_pp) == IS_LONG) {
@@ -247,6 +248,11 @@ PHP_METHOD(RedisArray, __construct)
}
}
}
+
+ /* extract lazy connect option. */
+ if(FAILURE != zend_hash_find(hOpts, "lazy_connect", sizeof("lazy_connect"), (void**)&zpData) && Z_TYPE_PP(zpData) == IS_BOOL) {
+ b_lazy_connect = Z_BVAL_PP(zpData);
+ }
}
/* extract either name of list of hosts from z0 */
@@ -256,7 +262,7 @@ PHP_METHOD(RedisArray, __construct)
break;
case IS_ARRAY:
- ra = ra_make_array(Z_ARRVAL_P(z0), z_fun, z_dist, hPrev, b_index, l_retry_interval TSRMLS_CC);
+ ra = ra_make_array(Z_ARRVAL_P(z0), z_fun, z_dist, hPrev, b_index, l_retry_interval, b_lazy_connect TSRMLS_CC);
break;
default:
View
107 redis_array_impl.c
@@ -29,7 +29,7 @@ extern int le_redis_sock;
extern zend_class_entry *redis_ce;
RedisArray*
-ra_load_hosts(RedisArray *ra, HashTable *hosts, long retry_interval TSRMLS_DC)
+ra_load_hosts(RedisArray *ra, HashTable *hosts, long retry_interval, zend_bool lazy_connect TSRMLS_DC)
{
int i, host_len, id;
int count = zend_hash_num_elements(hosts);
@@ -67,10 +67,13 @@ ra_load_hosts(RedisArray *ra, HashTable *hosts, long retry_interval TSRMLS_DC)
call_user_function(&redis_ce->function_table, &ra->redis[i], &z_cons, &z_ret, 0, NULL TSRMLS_CC);
/* create socket */
- redis_sock = redis_sock_create(host, host_len, port, 0, 0, NULL, retry_interval); /* TODO: persistence? */
+ redis_sock = redis_sock_create(host, host_len, port, 0, 0, NULL, retry_interval, lazy_connect); /* TODO: persistence? */
- /* connect */
- redis_sock_server_open(redis_sock, 1 TSRMLS_CC);
+ if (!lazy_connect)
+ {
+ /* connect */
+ redis_sock_server_open(redis_sock, 1 TSRMLS_CC);
+ }
/* attach */
#if PHP_VERSION_ID >= 50400
@@ -159,10 +162,12 @@ RedisArray *ra_load_array(const char *name TSRMLS_DC) {
zval *z_params_index;
zval *z_params_autorehash;
zval *z_params_retry_interval;
+ zval *z_params_lazy_connect;
RedisArray *ra = NULL;
zend_bool b_index = 0, b_autorehash = 0;
long l_retry_interval = 0;
+ zend_bool b_lazy_connect = 0;
HashTable *hHosts = NULL, *hPrev = NULL;
/* find entry */
@@ -240,8 +245,18 @@ RedisArray *ra_load_array(const char *name TSRMLS_DC) {
}
}
+ /* find retry interval option */
+ MAKE_STD_ZVAL(z_params_lazy_connect);
+ array_init(z_params_lazy_connect);
+ sapi_module.treat_data(PARSE_STRING, estrdup(INI_STR("redis.arrays.lazyconnect")), z_params_lazy_connect TSRMLS_CC);
+ if (zend_hash_find(Z_ARRVAL_P(z_params_lazy_connect), name, strlen(name) + 1, (void **) &z_data_pp) != FAILURE) {
+ if(Z_TYPE_PP(z_data_pp) == IS_STRING && strncmp(Z_STRVAL_PP(z_data_pp), "1", 1) == 0) {
+ b_lazy_connect = 1;
+ }
+ }
+
/* create RedisArray object */
- ra = ra_make_array(hHosts, z_fun, z_dist, hPrev, b_index, l_retry_interval TSRMLS_CC);
+ ra = ra_make_array(hHosts, z_fun, z_dist, hPrev, b_index, l_retry_interval, b_lazy_connect TSRMLS_CC);
ra->auto_rehash = b_autorehash;
/* cleanup */
@@ -257,12 +272,14 @@ RedisArray *ra_load_array(const char *name TSRMLS_DC) {
efree(z_params_autorehash);
zval_dtor(z_params_retry_interval);
efree(z_params_retry_interval);
+ zval_dtor(z_params_lazy_connect);
+ efree(z_params_lazy_connect);
return ra;
}
RedisArray *
-ra_make_array(HashTable *hosts, zval *z_fun, zval *z_dist, HashTable *hosts_prev, zend_bool b_index, long retry_interval TSRMLS_DC) {
+ra_make_array(HashTable *hosts, zval *z_fun, zval *z_dist, HashTable *hosts_prev, zend_bool b_index, long retry_interval, zend_bool lazy_connect TSRMLS_DC) {
int count = zend_hash_num_elements(hosts);
@@ -280,10 +297,10 @@ ra_make_array(HashTable *hosts, zval *z_fun, zval *z_dist, HashTable *hosts_prev
/* init array data structures */
ra_init_function_table(ra);
- if(NULL == ra_load_hosts(ra, hosts, retry_interval TSRMLS_CC)) {
+ if(NULL == ra_load_hosts(ra, hosts, retry_interval, lazy_connect TSRMLS_CC)) {
return NULL;
}
- ra->prev = hosts_prev ? ra_make_array(hosts_prev, z_fun, z_dist, NULL, b_index, retry_interval TSRMLS_CC) : NULL;
+ ra->prev = hosts_prev ? ra_make_array(hosts_prev, z_fun, z_dist, NULL, b_index, retry_interval, lazy_connect TSRMLS_CC) : NULL;
/* copy function if provided */
if(z_fun) {
@@ -395,12 +412,56 @@ ra_call_distributor(RedisArray *ra, const char *key, int key_len, int *pos TSRML
return 1;
}
+zend_bool
+ra_check_distributor(RedisArray *ra, int *num_original, int *num_reshards TSRMLS_DC) {
+ if(Z_TYPE_P(ra->z_dist) == IS_ARRAY) {
+ zval **z_original_pp;
+ zval **z_reshards_pp;
+ HashTable *shards = Z_ARRVAL_P(ra->z_dist);
+ if (zend_hash_num_elements(shards) != 2) {
+ return 0;
+ }
+ if (zend_hash_index_find(shards, 0, (void **)&z_original_pp) != SUCCESS ||
+ zend_hash_index_find(shards, 1, (void **)&z_reshards_pp) != SUCCESS) {
+ return 0;
+ }
+ if (Z_TYPE_PP(z_original_pp) == IS_LONG) {
+ if ((*num_original = Z_LVAL_PP(z_original_pp)) == 0) {
+ return 0;
+ }
+ }
+ else if (Z_TYPE_PP(z_original_pp) == IS_STRING) {
+ if ((*num_original = atol(Z_STRVAL_PP(z_original_pp))) == 0) {
+ return 0;
+ }
+ }
+ else {
+ return 0;
+ }
+ if (Z_TYPE_PP(z_reshards_pp) == IS_LONG) {
+ if ((*num_reshards = Z_LVAL_PP(z_reshards_pp)) == 0) {
+ return 0;
+ }
+ }
+ else if (Z_TYPE_PP(z_reshards_pp) == IS_STRING) {
+ if ((*num_reshards = atol(Z_STRVAL_PP(z_reshards_pp))) == 0) {
+ return 0;
+ }
+ }
+ else {
+ return 0;
+ }
+ return 1;
+ }
+ return 0;
+}
+
zval *
ra_find_node(RedisArray *ra, const char *key, int key_len, int *out_pos TSRMLS_DC) {
uint32_t hash;
char *out;
- int pos, out_len;
+ int pos = 0, out_len;
/* extract relevant part of the key */
out = ra_extract_key(ra, key, key_len, &out_len TSRMLS_CC);
@@ -408,7 +469,32 @@ ra_find_node(RedisArray *ra, const char *key, int key_len, int *out_pos TSRMLS_D
return NULL;
if(ra->z_dist) {
- if (!ra_call_distributor(ra, key, key_len, &pos TSRMLS_CC)) {
+ char *error = NULL;
+ int num_original, num_reshards;
+ if (ra_check_distributor(ra, &num_original, &num_reshards TSRMLS_CC)) {
+ if (num_reshards < 1 || ra->count != (num_original * (1 << num_reshards))) {
+ return NULL;
+ }
+ /* Calculate original hash */
+ hash = rcrc32(out, out_len);
+ efree(out);
+ uint64_t h64 = hash;
+ h64 *= num_original;
+ h64 /= 0xffffffff;
+ pos = (int)h64;
+ int i;
+ /* Infer the new position */
+ for(i = 0; i < num_reshards; i++) {
+ int total = num_original * 2;
+ h64 = hash;
+ h64 *= total;
+ h64 /= 0xffffffff;
+ h64 %= 2;
+ pos = pos + h64 * num_original;
+ num_original = total;
+ }
+ }
+ else if (!ra_call_distributor(ra, key, key_len, &pos TSRMLS_CC)) {
return NULL;
}
}
@@ -424,7 +510,6 @@ ra_find_node(RedisArray *ra, const char *key, int key_len, int *out_pos TSRMLS_D
pos = (int)h64;
}
if(out_pos) *out_pos = pos;
-
return ra->redis[pos];
}
View
4 redis_array_impl.h
@@ -5,9 +5,9 @@
#include "common.h"
#include "redis_array.h"
-RedisArray *ra_load_hosts(RedisArray *ra, HashTable *hosts, long retry_interval TSRMLS_DC);
+RedisArray *ra_load_hosts(RedisArray *ra, HashTable *hosts, long retry_interval, zend_bool lazy_connect TSRMLS_DC);
RedisArray *ra_load_array(const char *name TSRMLS_DC);
-RedisArray *ra_make_array(HashTable *hosts, zval *z_fun, zval *z_dist, HashTable *hosts_prev, zend_bool b_index, long retry_interval TSRMLS_DC);
+RedisArray *ra_make_array(HashTable *hosts, zval *z_fun, zval *z_dist, HashTable *hosts_prev, zend_bool b_index, long retry_interval, zend_bool lazy_connect TSRMLS_DC);
zval *ra_find_node_by_name(RedisArray *ra, const char *host, int host_len TSRMLS_DC);
zval *ra_find_node(RedisArray *ra, const char *key, int key_len, int *out_pos TSRMLS_DC);
void ra_init_function_table(RedisArray *ra);
View
4 redis_session.c
@@ -277,9 +277,9 @@ PS_OPEN_FUNC(redis)
RedisSock *redis_sock;
if(url->host) {
- redis_sock = redis_sock_create(url->host, strlen(url->host), url->port, timeout, persistent, persistent_id, retry_interval);
+ redis_sock = redis_sock_create(url->host, strlen(url->host), url->port, timeout, persistent, persistent_id, retry_interval, 0);
} else { /* unix */
- redis_sock = redis_sock_create(url->path, strlen(url->path), 0, timeout, persistent, persistent_id, retry_interval);
+ redis_sock = redis_sock_create(url->path, strlen(url->path), 0, timeout, persistent, persistent_id, retry_interval, 0);
}
redis_pool_add(pool, redis_sock, weight, database, prefix, auth TSRMLS_CC);
Something went wrong with that request. Please try again.