hmset returns server error: 'Protocol error: too big bulk count string'

[mikemccabe]

In our production system, I'm seeing occasional failures with hmsets with large arguments. For some items, the failure rate in production is 20% or so.

PHP Fatal error: Uncaught exception 'Predis\ServerException' with message 'ERR Protocol error: too big bulk count string' in phar:///predis_0.7.1.phar/Predis/Network/StreamConnection.php:257
Stack trace:

0 phar:///predis_0.7.1.phar/Predis/Network/ConnectionBase.php(142): Predis\Network\StreamConnection->read()

1 phar:///predis_0.7.1.phar/Predis/Network/ConnectionBase.php(134): Predis\Network\ConnectionBase->readResponse(Object(Predis\Commands\HashSetMultiple))

2 phar:///predis_0.7.1.phar/Predis/Client.php(219): Predis\Network\ConnectionBase->executeCommand(Object(Predis\Commands\HashSetMultiple))

3 [internal function]: Predis\Client->__call('hmset', Array)

4 bugtest.php(42): Predis\Client->hmset('bugtest', Array)

5 {main}

thrown in phar:///predis_0.7.1.phar/Predis/Network/StreamConnection.php on line 257

I've been able to reproduce it in a standalone test, but at a much lower rate - usually much less than 1%.

Test code to reproduce:

<?
require_once 'predis_0.7.1.phar';
for ($i = 0; $i < 10000; $i++) {
  if ($i % 100 == 0) {
    print "$i\n";
  }
  $redis = new Predis\Client('tcp://redis_server:6379');
  $a = array('f' => str_repeat('a', 74707),
             'r' => str_repeat('a', 2),
             'm' => str_repeat('a', 800));
  $redis->hmset('bugtest', $a);
  unset($redis);
}

Eliciting the bug seems to require a new client instance with each loop. I also couldn't reproduce the bug unless connecting to a Redis server on a different machine.

I see that the error string 'too big bulk count string' occurs in the Redis source (2.4.7), and is returned when this test in networking.c succeeds:

if (sdslen(c->querybuf) > REDIS_INLINE_MAX_SIZE) /* 1024 * 64 */

php --version:
PHP 5.3.2-1ubuntu4.7 with Suhosin-Patch (cli)

[mikemccabe]

I see the error more frequently with a shell loop running the following script:

<?
require_once 'predis_0.7.1.phar';
$redis = new Predis\Client('tcp://redis_server:6379');
$f = str_repeat('a', 74707);
$r = str_repeat('a', 2);
$m = str_repeat('a', 800);

$a = array('f' => $f,
           'r' => $r,
           'm' => $m);
$redis->hmset('bugtest', $a);

[nrk]

Hi @mikemccabe, sorry for the wait.

I'm still looking into this issue since I couldn't reproduce your case yet. Actually I'm not sure about what could trigger c->querybuf to grow outside the allowed size specified by REDIS_INLINE_MAX_SIZE in Redis, might end up asking @antirez for advice here.

[nrk]

It doesn't seem a bug of Predis and the same thing happens with other client libraries such as phpredis. It's most likely just the hard limit set for the maximum size of bulk values after all...

[piotr-piatkowski]

Just found this bug in redis server, fix is trivial and is included with the ticket, look here: redis/redis#673

[nrk]

@piotr-piatkowski thanks for the pointer, it's definitely helpful to have it linked here.