Obtained unexpected value after "Protocol error from client ..."

[keigokeigo]

Hi,

I got following error on redis server, on one of our cluster members.

Oct  5 13:26:22 redis-014 redis[16155]: Reading from client: Connection reset by peer
Oct  5 13:26:22 redis-014 redis[16155]: Protocol error from client: id=12525652 addr=172.20.82.37:49112 fd=1008 name= age=481 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=72771 qbuf-free=36929 obl=0 oll=4 omem=90432 events=rw cmd=NULL

I am not sure if some kind of network disconnection occurred or not. There was no evidence.

Once the server falls into such a situation, client side (=phpredis) works strangely like below.

[1] phpredis exports log lines like,

PHP Warning:  igbinary_unserialize_zval: unknown type '4b', position 14481 in /foo/bar/redis_cluster.php on line 777
PHP Warning: igbinary_unserialize_array: unknown key type '35', position 17352 in /foo/bar/redis_cluster.php on line 777

(*) Note: line 777 is the line where RedisCluster:get() is being called.

PHP Warning: RedisCluster::del(): Invalid reply type returned for DEL command in /foo/bar/redis_cluster.php on line 555
(*) Note: 777 is the line where RedisCluster:del() is being called.

[2] Obtained incorrect value.
For example,

• Correct cache data (key => value pair)
  "member_id:999" => { "name" : "Bar", "address" : "Baz" }
• But obtained value was
  "member_id:999" => { "category_id" : 999, "category_name" : "Sport" }

It seems phpredis obtained a value of another key.

Only one way to resolve above problem is restarting httpd (Apache) process.
After that, phpredis works properly until next phenomenon happens.

Are there any solution for this?

[nicofff]

Hey @keigokeigo,
Did you find a solution to this?

I'm seeing similar errors with either persistance or distributing reads to the slaves.
Disabling both seems to be a lot more stable

[keigokeigo]

@nicofff
Thank you for your reply.

Did you find a solution to this?

Unfortunately we are still struggling with it.
But we made some changes below to reduce such situation.
It's not been completely resolved yet.

[1] Apache setting MaxRequestsPerChild

// Before
MaxRequestsPerChild 0
// After
MaxRequestsPerChild 10000

I assumed some memory corruption is happening under the situation.
So I gave a life cycle to Apache forked child processes.
That enables child process to restart about every 5 minutes.

It worked and it got even better than before.
But it still sometimes happens that phpredis obtains different value for a key and our web site publishes wired contents. After that we only have to restart Apache parent process.

[2] Reducing access to redis.
We made some changes to php web application to reduce access to redis servers.
Now frequently accessed cache objects is stored on redis-client side (web application itself) using APCu user cache.
As a result traffic and TCP session to redis is significantly reduced than before.
But the problem, obtaining different value, is not resolved at all. I am not sure it worked and got better.

I'm seeing similar errors with either persistance or distributing reads to the slaves.
Disabling both seems to be a lot more stable

We are running a large scale web site.
So persistance connection to redis is absolutely required not to consume ephemeral port (client port).
We can't disable it.

We've already disabled distributing reads to the slaves.
We don't set any options (=we don't call any setOption() method) to read slaves but follow default options to read only from master nodes.
https://github.com/phpredis/phpredis/blob/feature/redis_cluster/cluster.markdown#automatic-slave-failover--distribution

// The default option, only send commands to master nodes
$obj_cluster->setOption(RedisCluster::OPT_FAILOVER, RedisCluster::FAILOVER_NONE);

Is it hard for you to fix the problem on your side?
We will definitely cooperate you to solve this problem.

FYI: Our system structure

phpredis
--> redis01 --(repl)-->redis11
--> redis02 --(repl)-->redis12
--> redis03 --(repl)-->redis13
--> redis04 --(repl)-->redis14
--> redis05 --(repl)-->redis15
--> redis06 --(repl)-->redis16
--> redis07 --(repl)-->redis17
--> redis08 --(repl)-->redis18
--> redis09 --(repl)-->redis19
--> redis10 --(repl)-->redis20

[nicofff]

hi @keigokeigo. I'm not a phpredis dev, I'm just another fellow admin in distress.

Out of curiosity, what PHP and phpredis versions are you using?

[keigokeigo]

Hi, @nicofff .

I'm not a phpredis dev, I'm just another fellow admin in distress.

Sorry about that.

The versions are,

• php-5.4.34
• php-pecl-igbinary-1.2.1
• php-redis-2.2.8
• php-pecl-zendopcache-7.0.5
• php-pecl-apcu-4.0.11

Hope above information helps you.

BTW,
I found a similar issue you posted "Wrong key is read (among other errors) · Issue #1080".
I think it is deeply linked with here. Let me keep track with it.

Thanks.

[ppanphper]

I also encountered the same problem, is there a solution?

[lokechenlin]

I also encountered the same problem, is there a solution?

[keigokeigo]

Hi, @ppanphper , @lokechenlin :

This topic is switched to another issue, "Wrong key is read (among other errors) · Issue #1080".
But it seems it is not in progress there.

Only one workaround I can tell you is 2 ways as I told before.
[1] Apache setting MaxRequestsPerChild
[2] Reducing access to redis.
You can reduce the problem but can't resolve completely.

I finally gave up phpredis and replaced to predis (with hiredis).
"hiredis" is official connection library so I expected it's stable.

I just replaced it and it's working fine so far.

[ppanphper]

@keigokeigo
Thank you for your reply.
My problem is "RedisCluster::del(): Invalid reply type returned for DEL command".

[keigokeigo]

Hi, @ppanphper

My problem is "RedisCluster::del(): Invalid reply type returned for DEL command".

I was also suffering from it.
Same as @nicoff in #1080 .

[yatsukhnenko]

Old issue without any activity for a long time. Feel free to open new issues if you have any questions about phpredis.