'read error on connection' #492

Closed
info2000 opened this Issue Aug 3, 2014 · 17 comments

Comments

Projects
None yet
9 participants
@info2000

info2000 commented Aug 3, 2014

Hi, how can solve this error?
PHP message: PHP Fatal error: Uncaught exception 'RedisException' with message 'read error on connection'
i'm getting it for this call: Redis->hGet('fechainicio', '1')

My system is:
PHP Version 5.4.4-14+
in phpinfo redis version 2.2.5 but in redis-cli redis_version:2.6.17
with 17 clients connected, take 6 seconds to show the command prompt when type redis-cli

Memory
used_memory:91931192
used_memory_human:87.67M
used_memory_rss:105947136
used_memory_peak:5178554456
used_memory_peak_human:4.82G
used_memory_lua:31744
mem_fragmentation_ratio:1.15
mem_allocator:jemalloc-3.2.0

Not save to disk
Thanks

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Aug 3, 2014

Member

My guess is you're hitting a timeout, either because Redis is under heavy load and/or paging, or because of network issues?

Have you tried running:

redis-cli -h <host> -p <port> --latency

That will let you know if you're having latency issues.

Member

michael-grunder commented Aug 3, 2014

My guess is you're hitting a timeout, either because Redis is under heavy load and/or paging, or because of network issues?

Have you tried running:

redis-cli -h <host> -p <port> --latency

That will let you know if you're having latency issues.

@info2000

This comment has been minimized.

Show comment
Hide comment
@info2000

info2000 Aug 3, 2014

min: 1, max: 4762, avg: 580.68 (81 samples)

info2000 commented Aug 3, 2014

min: 1, max: 4762, avg: 580.68 (81 samples)

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Aug 5, 2014

Member

To me this looks like a timeout failure. A max latency of 4762ms is quite high, average 580 is also quite high. Is the server under tremendous load or paging to disk?

Member

michael-grunder commented Aug 5, 2014

To me this looks like a timeout failure. A max latency of 4762ms is quite high, average 580 is also quite high. Is the server under tremendous load or paging to disk?

@info2000

This comment has been minimized.

Show comment
Hide comment
@info2000

info2000 Aug 5, 2014

The server only has 3k http requests per minute, the 99% of their has this schema:
nginx=>php=>echo get content from redis, some atomic increment in redis
the other 1% of requests call to database and store result in redis
Server has 64gb of ram
About paging to disk, if refer to save redis to disk, i have disable it, if no refer to it, explain it please
Thanks

info2000 commented Aug 5, 2014

The server only has 3k http requests per minute, the 99% of their has this schema:
nginx=>php=>echo get content from redis, some atomic increment in redis
the other 1% of requests call to database and store result in redis
Server has 64gb of ram
About paging to disk, if refer to save redis to disk, i have disable it, if no refer to it, explain it please
Thanks

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Aug 5, 2014

Member

Hey,

It's difficult for me to diagnose the problem without knowing more information. These latency numbers are really, really high:

min: 1, max: 4762, avg: 580.68 (81 samples)

Here is a random production server we use, which is under constant load (>50k ops/sec):

min: 0, max: 4, avg: 0.69 (898 samples)

The most likely possibilities:

  1. You have a network issue. You could test that by running --latency on the server itself (assuming you didn't).
  2. You have operations that are incredibly expensive (e.g. KEYS, ZUNIONSTORE on huge sets, etc)
  3. The server is out of ram and paging. Honestly though, Redis is still generally fast in this case as long as you aren't doing expensive ops.
Member

michael-grunder commented Aug 5, 2014

Hey,

It's difficult for me to diagnose the problem without knowing more information. These latency numbers are really, really high:

min: 1, max: 4762, avg: 580.68 (81 samples)

Here is a random production server we use, which is under constant load (>50k ops/sec):

min: 0, max: 4, avg: 0.69 (898 samples)

The most likely possibilities:

  1. You have a network issue. You could test that by running --latency on the server itself (assuming you didn't).
  2. You have operations that are incredibly expensive (e.g. KEYS, ZUNIONSTORE on huge sets, etc)
  3. The server is out of ram and paging. Honestly though, Redis is still generally fast in this case as long as you aren't doing expensive ops.
@info2000

This comment has been minimized.

Show comment
Hide comment
@info2000

info2000 Aug 5, 2014

Tell me what more info need
network issue: the test was did in localhost
operations: i only use: hget, hset, lpop, and hincr
The redis is set with 100 mb of memory

info2000 commented Aug 5, 2014

Tell me what more info need
network issue: the test was did in localhost
operations: i only use: hget, hset, lpop, and hincr
The redis is set with 100 mb of memory

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Aug 5, 2014

Member

I'm happy to try and help, but this doesn't appear to be related to phpredis at all. 😄

Could you send me the output from 👍

redis info
and
slowlog get 10

Cheers
Mike

Member

michael-grunder commented Aug 5, 2014

I'm happy to try and help, but this doesn't appear to be related to phpredis at all. 😄

Could you send me the output from 👍

redis info
and
slowlog get 10

Cheers
Mike

@info2000

This comment has been minimized.

Show comment
Hide comment
@info2000

info2000 Aug 5, 2014

This is the data of redis-info, the server is recently restarted, i'm also getting high cpu usage in the php pool that call to redis.
Slow log return empty list

Server

redis_version:2.6.17
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 3.10.23-xxxx-std-ipv6-64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.7.2
process_id:5107
run_id:74a23fe027297315ab832f7483326d057f29785b
tcp_port:6379
uptime_in_seconds:406
uptime_in_days:0
hz:10
lru_clock:218698

Clients

connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:47619360
used_memory_human:45.41M
used_memory_rss:49885184
used_memory_peak:5178553432
used_memory_peak_human:4.82G
used_memory_lua:31744
mem_fragmentation_ratio:1.05
mem_allocator:jemalloc-3.2.0

Persistence

loading:0
rdb_changes_since_last_save:195548
rdb_bgsave_in_progress:0
rdb_last_save_time:1407278415
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

Stats

total_connections_received:39364
total_commands_processed:313049
instantaneous_ops_per_sec:1053
rejected_connections:0
expired_keys:0
evicted_keys:764
keyspace_hits:0
keyspace_misses:78610
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

Replication

role:master
connected_slaves:0

CPU

used_cpu_sys:10.82
used_cpu_user:25.61
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Keyspace

db0:keys=116,expires=0,avg_ttl=0

info2000 commented Aug 5, 2014

This is the data of redis-info, the server is recently restarted, i'm also getting high cpu usage in the php pool that call to redis.
Slow log return empty list

Server

redis_version:2.6.17
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 3.10.23-xxxx-std-ipv6-64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.7.2
process_id:5107
run_id:74a23fe027297315ab832f7483326d057f29785b
tcp_port:6379
uptime_in_seconds:406
uptime_in_days:0
hz:10
lru_clock:218698

Clients

connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:47619360
used_memory_human:45.41M
used_memory_rss:49885184
used_memory_peak:5178553432
used_memory_peak_human:4.82G
used_memory_lua:31744
mem_fragmentation_ratio:1.05
mem_allocator:jemalloc-3.2.0

Persistence

loading:0
rdb_changes_since_last_save:195548
rdb_bgsave_in_progress:0
rdb_last_save_time:1407278415
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

Stats

total_connections_received:39364
total_commands_processed:313049
instantaneous_ops_per_sec:1053
rejected_connections:0
expired_keys:0
evicted_keys:764
keyspace_hits:0
keyspace_misses:78610
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

Replication

role:master
connected_slaves:0

CPU

used_cpu_sys:10.82
used_cpu_user:25.61
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Keyspace

db0:keys=116,expires=0,avg_ttl=0

@folkenfanel

This comment has been minimized.

Show comment
Hide comment
@folkenfanel

folkenfanel Aug 14, 2014

Got the same error with $redis->subscribe(array($channel), 'callback'); The weird thing is, the callback function was fired.

Got the same error with $redis->subscribe(array($channel), 'callback'); The weird thing is, the callback function was fired.

@andygrunwald

This comment has been minimized.

Show comment
Hide comment
@andygrunwald

andygrunwald Aug 20, 2014

This error is really hard to track.
Many informations are collected in #70

This error is really hard to track.
Many informations are collected in #70

@michael-grunder

This comment has been minimized.

Show comment
Hide comment
@michael-grunder

michael-grunder Aug 20, 2014

Member

It is indeed. We actually had this error ourselves when we first started using phpredis (way before I ever made a commit into the codebase), and got rid of it by tuning our timeouts and things like TCP_WAIT settings, etc.

We still get the error maybe once a month, but that's when we do something silly and request a ZUNIONSTORE on a million sets, and actually are encountering a timeout.

As @andygrunwald said, there is a lot of info in #70

Cheers guys!
Mike

Member

michael-grunder commented Aug 20, 2014

It is indeed. We actually had this error ourselves when we first started using phpredis (way before I ever made a commit into the codebase), and got rid of it by tuning our timeouts and things like TCP_WAIT settings, etc.

We still get the error maybe once a month, but that's when we do something silly and request a ZUNIONSTORE on a million sets, and actually are encountering a timeout.

As @andygrunwald said, there is a lot of info in #70

Cheers guys!
Mike

@andygrunwald

This comment has been minimized.

Show comment
Hide comment
@andygrunwald

andygrunwald May 29, 2015

Should we close this in case of duplicate of #70 ?

Should we close this in case of duplicate of #70 ?

@Evandar276

This comment has been minimized.

Show comment
Hide comment
@Evandar276

Evandar276 Feb 13, 2017

I have just discovered that this error pops out when I send redis request after this: set_time_limit(600);

I have redis running normally, but this breaks it every time. Is it correct behavior, or bug in phpredis?

I have just discovered that this error pops out when I send redis request after this: set_time_limit(600);

I have redis running normally, but this breaks it every time. Is it correct behavior, or bug in phpredis?

@jgimeno

This comment has been minimized.

Show comment
Hide comment
@jgimeno

jgimeno Feb 24, 2017

I am having this same problem, as @Evandar276 is saying. Is this an issue with the module?

jgimeno commented Feb 24, 2017

I am having this same problem, as @Evandar276 is saying. Is this an issue with the module?

@muayyad-alsadi

This comment has been minimized.

Show comment
Hide comment
@muayyad-alsadi

muayyad-alsadi Jul 5, 2017

Got the same error with $redis->subscribe(array($channel), 'callback'); The weird thing is, the callback function was fired.

@folkenfanel this was the solution from our side (php)

      ini_set("default_socket_timeout", -1);

Got the same error with $redis->subscribe(array($channel), 'callback'); The weird thing is, the callback function was fired.

@folkenfanel this was the solution from our side (php)

      ini_set("default_socket_timeout", -1);
@Devying

This comment has been minimized.

Show comment
Hide comment
@Devying

Devying Aug 15, 2017

set redis connect timeout=0 and
ini_set("default_socket_timeout", -1);

Devying commented Aug 15, 2017

set redis connect timeout=0 and
ini_set("default_socket_timeout", -1);

@yatsukhnenko

This comment has been minimized.

Show comment
Hide comment
@yatsukhnenko

yatsukhnenko Oct 19, 2017

Member

Duplicate #70

Member

yatsukhnenko commented Oct 19, 2017

Duplicate #70

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment