Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tune experiments dying due to redis crashes (Ray Redis version) #2028

Closed
nikhilkhanna opened this issue May 10, 2018 · 3 comments
Closed

tune experiments dying due to redis crashes (Ray Redis version) #2028

nikhilkhanna opened this issue May 10, 2018 · 3 comments

Comments

@nikhilkhanna
Copy link

nikhilkhanna commented May 10, 2018 •
edited
Loading

System information

  • Linux Ubuntu 16.04)
  • Ray install command: pip install ray
  • Ray version: 0.4.0
  • Python version: python 2.7:

Describe the problem

While executing long-running training jobs via ray tune, jobs will occassionaly (though quite frequently) die during execution. Checking out the logs in /tmp/raylogs the monitor-*.err log on ray-head shows
that there was a redis connection error and looking at the redis output we can see that it crashed "Redis 3.9.102 crashed by signal: 11".

I noticed in crash report the version of Redis is 3.9.102 which is an older version of redis, the current version is 4.0.9. Checking old redis github issues https://github.com/antirez/redis/issues it seems that this is an error that has been fixed with / since the release of Redis 4.0. Is this old version of redis that is a result of ray's configuration? I didn't have redis installed previously on this machine. If this is a result of ray's configuration, is there any reason ray isn't using the latest version of redis in order to get the most recent bug fixes to fix problems like this?

Source code / logs

Here is the log on the monitor-*.err file:

File "/usr/local/lib/python2.7/dist-packages/ray/monitor.py", line 597, in
monitor.run()
File "/usr/local/lib/python2.7/dist-packages/ray/monitor.py", line 520, in run
local_schedulers = self.state.local_schedulers()
File "/usr/local/lib/python2.7/dist-packages/ray/experimental/state.py", line 876, in local_schedulers
clients = self.client_table()
File "/usr/local/lib/python2.7/dist-packages/ray/experimental/state.py", line 351, in client_table
db_client_keys = self.redis_client.keys(DB_CLIENT_PREFIX + "*")
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 1032, in keys
return self.execute_command('KEYS', pattern)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 674, in execute_command
return self.parse_response(connection, command_name, **options)
File "/usr/local/lib/python2.7/dist-packages/redis/client.py", line 680, in parse_response
response = connection.read_response()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 624, in read_response
response = self._parser.read_response()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 284, in read_response
response = self._buffer.readline()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 216, in readline
self._read_from_socket()
File "/usr/local/lib/python2.7/dist-packages/redis/connection.py", line 191, in _read_from_socket
(e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: (104, 'Connection reset by peer')

Here is the redis bug report below given by

=== REDIS BUG REPORT START: Cut & paste starting from here ===
22:M 10 May 01:18:59.743 # Redis 3.9.102 crashed by signal: 11
22:M 10 May 01:18:59.743 # Crashed running the instuction at: 0x489914
22:M 10 May 01:18:59.743 # Accessing address: 0x8
22:M 10 May 01:18:59.743 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(RM_StringToLongLong+0x4)[0x489914]

Backtrace:
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(logStackTrace+0x29)[0x464a39]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(sigsegvHandler+0xac)[0x4650ec]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fc42a868890]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(RM_StringToLongLong+0x4)[0x489914]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/redis_module/libray_redis_module.so(_Z23Disconnect_RedisCommandP14RedisModuleCtxPP17RedisModuleStringi+0x81)[0x7fc4285ed941]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(RedisModuleCommandDispatcher+0x3f)[0x48b3ff]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(call+0x9e)[0x42a28e]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(processCommand+0x367)[0x42a947]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(processInputBuffer+0x105)[0x439885]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(aeProcessEvents+0x218)[0x424c28]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(aeMain+0x2b)[0x424edb]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379(main+0x41d)[0x421ead]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7fc42a486b97]
/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379[0x422125]

------ INFO OUTPUT ------

Server

redis_version:3.9.102
redis_git_sha1:f69cbd35
redis_git_dirty:0
redis_build_id:ffc7171ab2fa3a3e
redis_mode:standalone
os:Linux 4.4.111-k8s x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:22
run_id:ffeb43370d301d52f92e7b417e211f0ab84f36c7
tcp_port:6379
uptime_in_seconds:7696
uptime_in_days:0
hz:10
lru_clock:15965699
executable:/usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server
config_file:

Clients

connected_clients:177
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:95651992
used_memory_human:91.22M
used_memory_rss:67866624
used_memory_rss_human:64.72M
used_memory_peak:95939984
used_memory_peak_human:91.50M
used_memory_peak_perc:99.70%
used_memory_overhead:4338312
used_memory_startup:765928
used_memory_dataset:91313680
used_memory_dataset_perc:96.24%
total_system_memory:16037273600
total_system_memory_human:14.94G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:0.71
mem_allocator:jemalloc-4.0.3
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:263110
rdb_bgsave_in_progress:0
rdb_last_save_time:1525911998
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

Stats

total_connections_received:9536
total_commands_processed:1730886
instantaneous_ops_per_sec:198
total_net_input_bytes:265723629
total_net_output_bytes:483818456
instantaneous_input_kbps:50.79
instantaneous_output_kbps:46.67
rejected_connections:7
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:378943
keyspace_misses:41801
pubsub_channels:7
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0

Replication

role:master
connected_slaves:0
master_replid:9fd690ad647472b6f0b70db729a4404956c28ae0
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:84.67
used_cpu_user:28.82
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Commandstats

cmdstat_get:calls=78,usec=176,usec_per_call=2.26
cmdstat_set:calls=10,usec=42,usec_per_call=4.20
cmdstat_setnx:calls=23,usec=82,usec_per_call=3.57
cmdstat_exists:calls=1,usec=1,usec_per_call=1.00
cmdstat_rpush:calls=8852,usec=288623,usec_per_call=32.61
cmdstat_llen:calls=232,usec=263,usec_per_call=1.13
cmdstat_lindex:calls=232,usec=537,usec_per_call=2.31
cmdstat_lrange:calls=61405,usec=190367,usec_per_call=3.10
cmdstat_zadd:calls=6785,usec=122753,usec_per_call=18.09
cmdstat_hget:calls=1,usec=3,usec_per_call=3.00
cmdstat_hmset:calls=34,usec=290,usec_per_call=8.53
cmdstat_hmget:calls=258,usec=735,usec_per_call=2.85
cmdstat_hgetall:calls=358538,usec=1857693,usec_per_call=5.18
cmdstat_keys:calls=70902,usec=1541840,usec_per_call=21.75
cmdstat_save:calls=4,usec=12738,usec_per_call=3184.50
cmdstat_flushall:calls=2,usec=5573,usec_per_call=2786.50
cmdstat_info:calls=3,usec=346,usec_per_call=115.33
cmdstat_config:calls=63,usec=16559,usec_per_call=262.84
cmdstat_subscribe:calls=49,usec=130,usec_per_call=2.65
cmdstat_publish:calls=1223387,usec=3793268,usec_per_call=3.10
cmdstat_client:calls=8,usec=1313,usec_per_call=164.12
cmdstat_command:calls=2,usec=840,usec_per_call=420.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=28,expires=0,avg_ttl=0
hash_init_value: 1525939682

------ CLIENT LIST OUTPUT ------
id=260 addr=100.96.13.131:45008 fd=146 name= age=7690 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=261 addr=100.96.13.131:45010 fd=147 name= age=7690 idle=7628 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=262 addr=100.96.12.151:42164 fd=148 name= age=7690 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=173 addr=100.96.12.151:42146 fd=99 name= age=7691 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=174 addr=100.96.12.151:42154 fd=100 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=175 addr=100.96.12.151:42156 fd=101 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=335 addr=100.96.14.190:49304 fd=25 name= age=7677 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=176 addr=100.96.12.151:42158 fd=102 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=150 addr=100.96.13.131:44954 fd=88 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=151 addr=100.96.13.131:44956 fd=89 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=152 addr=100.96.13.131:44958 fd=90 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=157 addr=100.96.11.153:36958 fd=91 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=272 addr=100.96.14.190:49252 fd=154 name= age=7689 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=275 addr=100.96.14.190:49258 fd=155 name= age=7689 idle=7689 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=159 addr=100.96.11.153:36960 fd=92 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish
id=160 addr=100.96.11.153:36962 fd=93 name= age=7691 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=161 addr=100.96.11.153:36970 fd=94 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=235 addr=100.96.13.131:44992 fd=130 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=256 addr=100.96.11.153:37002 fd=144 name= age=7690 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=238 addr=100.96.13.131:44998 fd=131 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=239 addr=100.96.11.153:36992 fd=132 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=242 addr=100.96.11.153:36998 fd=133 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=243 addr=100.96.9.115:38988 fd=134 name= age=7690 idle=7622 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=244 addr=100.96.9.115:38990 fd=135 name= age=7690 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=245 addr=100.96.13.131:45000 fd=136 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=267 addr=100.96.12.151:42174 fd=151 name= age=7689 idle=7689 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=268 addr=100.96.12.151:42176 fd=152 name= age=7689 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=324 addr=100.96.12.152:59642 fd=182 name= age=7685 idle=3266 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=271 addr=100.96.12.151:42182 fd=153 name= age=7689 idle=7689 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=289 addr=100.96.12.152:59570 fd=164 name= age=7687 idle=7612 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=291 addr=100.96.12.152:59572 fd=165 name= age=7687 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=292 addr=100.96.12.152:59574 fd=166 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=293 addr=100.96.12.152:59582 fd=167 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=294 addr=100.96.12.152:59584 fd=168 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=295 addr=100.96.12.152:59586 fd=169 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=247 addr=100.96.14.190:49228 fd=138 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=250 addr=100.96.14.190:49230 fd=137 name= age=7690 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd
id=251 addr=100.96.14.190:49232 fd=139 name= age=7690 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=252 addr=100.96.13.131:45006 fd=140 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=253 addr=100.96.14.190:49240 fd=141 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=254 addr=100.96.14.190:49242 fd=142 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=255 addr=100.96.14.190:49244 fd=143 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=162 addr=100.96.11.153:36972 fd=95 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=163 addr=100.96.11.153:36974 fd=96 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=323 addr=100.96.12.152:59640 fd=181 name= age=7685 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=125 addr=100.96.13.131:44936 fd=72 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=128 addr=100.96.12.151:42116 fd=73 name= age=7691 idle=7676 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=130 addr=100.96.12.151:42118 fd=74 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=131 addr=100.96.12.151:42120 fd=75 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=285 addr=100.96.12.152:59562 fd=162 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=286 addr=100.96.12.152:59564 fd=163 name= age=7687 idle=7687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=264 addr=100.96.12.151:42168 fd=150 name= age=7689 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=132 addr=100.96.12.151:42128 fd=76 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=133 addr=100.96.12.151:42130 fd=77 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=134 addr=100.96.12.151:42132 fd=78 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=227 addr=100.96.11.153:36984 fd=128 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=230 addr=100.96.11.153:36990 fd=129 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=212 addr=100.96.9.115:38972 fd=118 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=322 addr=100.96.12.152:59638 fd=180 name= age=7685 idle=7685 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=213 addr=100.96.9.115:38973 fd=119 name= age=7690 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=339 addr=100.96.14.190:49316 fd=188 name= age=7677 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=218 addr=100.96.9.115:38984 fd=120 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=219 addr=100.96.9.115:38986 fd=121 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=220 addr=100.96.14.190:49202 fd=122 name= age=7690 idle=7674 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=222 addr=100.96.14.190:49204 fd=123 name= age=7690 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=223 addr=100.96.14.190:49206 fd=124 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=224 addr=100.96.14.190:49214 fd=125 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=225 addr=100.96.14.190:49216 fd=126 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=226 addr=100.96.14.190:49218 fd=127 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=23 addr=100.96.14.189:56754 fd=11 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=95 addr=100.96.11.153:36924 fd=54 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=96 addr=100.96.11.153:36926 fd=55 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=99 addr=100.96.10.5:34264 fd=56 name= age=7691 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=100 addr=100.96.9.115:38922 fd=57 name= age=7691 idle=7678 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=319 addr=100.96.12.152:59632 fd=179 name= age=7685 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=24 addr=100.96.14.189:56756 fd=12 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=208 addr=100.96.14.190:49194 fd=116 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=209 addr=100.96.14.190:49196 fd=117 name= age=7690 idle=7690 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=90 addr=100.96.9.115:38914 fd=52 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=91 addr=100.96.9.115:38916 fd=53 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=38 addr=100.96.14.189:56788 fd=19 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=63 addr=100.96.14.189:56846 fd=37 name= age=7693 idle=7631 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=80 addr=100.96.10.5:34242 fd=46 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=82 addr=100.96.10.5:34244 fd=47 name= age=7692 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=83 addr=100.96.10.5:34246 fd=48 name= age=7692 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=84 addr=100.96.10.5:34254 fd=49 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=85 addr=100.96.10.5:34256 fd=50 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=86 addr=100.96.10.5:34258 fd=51 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=315 addr=100.96.12.152:59624 fd=177 name= age=7685 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=318 addr=100.96.12.152:59630 fd=178 name= age=7685 idle=7685 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=40 addr=100.96.14.189:56790 fd=20 name= age=7696 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=263 addr=100.96.14.190:49250 fd=149 name= age=7689 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=52 addr=100.96.14.189:56820 fd=30 name= age=7694 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=keys
id=53 addr=100.96.14.189:56822 fd=31 name= age=7694 idle=0 flags=N db=0 sub=4 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=41 addr=100.96.14.189:56792 fd=21 name= age=7696 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=69 addr=100.96.10.5:34216 fd=40 name= age=7692 idle=7258 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=71 addr=100.96.10.5:34218 fd=41 name= age=7692 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=72 addr=100.96.10.5:34220 fd=42 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=73 addr=100.96.10.5:34228 fd=43 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=74 addr=100.96.10.5:34230 fd=44 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=75 addr=100.96.10.5:34232 fd=45 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=54 addr=100.96.14.189:56828 fd=32 name= age=7694 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=57 addr=100.96.14.189:56834 fd=33 name= age=7694 idle=7694 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=42 addr=100.96.14.189:56800 fd=22 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=65 addr=100.96.10.5:34208 fd=38 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=314 addr=100.96.12.152:59618 fd=176 name= age=7686 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=66 addr=100.96.10.5:34210 fd=39 name= age=7692 idle=7692 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=43 addr=100.96.14.189:56802 fd=23 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=44 addr=100.96.14.189:56804 fd=24 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=13 addr=100.96.14.189:56722 fd=6 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=18 addr=100.96.14.189:56740 fd=7 name= age=7696 idle=7674 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=20 addr=100.96.14.189:56742 fd=8 name= age=7696 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ray.disconnect
id=205 addr=100.96.10.5:34282 fd=114 name= age=7690 idle=7640 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=206 addr=100.96.10.5:34284 fd=115 name= age=7690 idle=7624 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=21 addr=100.96.14.189:56744 fd=9 name= age=7696 idle=0 flags=N db=0 sub=2 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=59 addr=100.96.14.189:56838 fd=35 name= age=7693 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=62 addr=100.96.14.189:56844 fd=36 name= age=7693 idle=7693 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=27 addr=100.96.14.189:56762 fd=13 name= age=7696 idle=7675 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=51 addr=100.96.14.189:56816 fd=29 name= age=7695 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=29 addr=100.96.14.189:56764 fd=14 name= age=7696 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=199 addr=100.96.13.131:44990 fd=110 name= age=7691 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=200 addr=100.96.10.5:34274 fd=111 name= age=7691 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=203 addr=100.96.10.5:34280 fd=112 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=204 addr=100.96.11.153:36980 fd=113 name= age=7691 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=30 addr=100.96.14.189:56766 fd=15 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=58 addr=100.96.14.189:56836 fd=34 name= age=7693 idle=7627 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=31 addr=100.96.14.189:56774 fd=16 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=32 addr=100.96.14.189:56776 fd=17 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=33 addr=100.96.14.189:56778 fd=18 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=22 addr=100.96.14.189:56752 fd=10 name= age=7696 idle=7696 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=103 addr=100.96.9.115:38924 fd=59 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=104 addr=100.96.9.115:38926 fd=60 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=282 addr=100.96.14.190:49272 fd=160 name= age=7688 idle=7612 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=283 addr=100.96.12.151:42188 fd=161 name= age=7688 idle=7624 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=195 addr=100.96.10.5:34266 fd=28 name= age=7691 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=198 addr=100.96.10.5:34272 fd=109 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=105 addr=100.96.12.151:42106 fd=58 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=106 addr=100.96.9.115:38934 fd=61 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=326 addr=100.96.12.152:59646 fd=183 name= age=7683 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=327 addr=100.96.12.152:59648 fd=184 name= age=7683 idle=7683 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=328 addr=100.96.12.152:59652 fd=185 name= age=7683 idle=3266 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hget
id=329 addr=100.96.12.152:59654 fd=186 name= age=7683 idle=7680 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=330 addr=100.96.12.152:59656 fd=187 name= age=7683 idle=3266 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=107 addr=100.96.9.115:38936 fd=62 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=108 addr=100.96.9.115:38938 fd=63 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=109 addr=100.96.12.151:42108 fd=64 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=124 addr=100.96.13.131:44934 fd=71 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=139 addr=100.96.9.115:38948 fd=79 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=141 addr=100.96.9.115:38950 fd=80 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd
id=194 addr=100.96.9.115:38970 fd=27 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lrange
id=142 addr=100.96.9.115:38952 fd=81 name= age=7691 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=280 addr=100.96.14.190:49270 fd=158 name= age=7688 idle=7617 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=281 addr=100.96.12.151:42186 fd=159 name= age=7688 idle=7628 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget
id=143 addr=100.96.9.115:38960 fd=82 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=144 addr=100.96.9.115:38962 fd=83 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=305 addr=100.96.12.152:59596 fd=170 name= age=7686 idle=7686 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=307 addr=100.96.12.152:59598 fd=171 name= age=7686 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=308 addr=100.96.12.152:59600 fd=172 name= age=7686 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=309 addr=100.96.12.152:59608 fd=173 name= age=7686 idle=7686 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=310 addr=100.96.12.152:59610 fd=174 name= age=7686 idle=7686 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=311 addr=100.96.12.152:59612 fd=175 name= age=7686 idle=7686 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=145 addr=100.96.9.115:38964 fd=84 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=179 addr=100.96.13.131:44968 fd=103 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=181 addr=100.96.13.131:44970 fd=104 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=zadd
id=182 addr=100.96.13.131:44972 fd=105 name= age=7691 idle=7674 flags=N db=0 sub=3 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=338 addr=100.96.14.190:49310 fd=26 name= age=7677 idle=7677 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=183 addr=100.96.13.131:44980 fd=106 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=184 addr=100.96.13.131:44982 fd=107 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=185 addr=100.96.13.131:44984 fd=108 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=146 addr=100.96.13.131:44942 fd=85 name= age=7691 idle=7678 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=148 addr=100.96.13.131:44944 fd=86 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=149 addr=100.96.13.131:44946 fd=87 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=170 addr=100.96.12.151:42142 fd=97 name= age=7691 idle=7691 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hgetall
id=276 addr=100.96.14.190:49262 fd=156 name= age=7689 idle=7640 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=279 addr=100.96.14.190:49268 fd=157 name= age=7689 idle=7689 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrange
id=172 addr=100.96.12.151:42144 fd=98 name= age=7691 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish
id=259 addr=100.96.11.153:37004 fd=145 name= age=7690 idle=7639 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hmget

------ CURRENT CLIENT INFO ------
id=20 addr=100.96.14.189:56742 fd=8 name= age=7696 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ray.disconnect
argv[0]: 'RAY.DISCONNECT'
argv[1]: 'd���'

------ REGISTERS ------
22:M 10 May 01:18:59.744 #
RAX:00007fc4287fcce8 RBX:00007fff173dcf90
RCX:00007fc42a2000c0 RDX:0000000000000028
RDI:0000000000000000 RSI:00007fff173dcf38
RBP:00007fc42a023180 RSP:00007fff173dcf10
R8 :0000000000000055 R9 :0000000000000054
R10:00007fc42a00dbe0 R11:0000000000000059
R12:0000000000000000 R13:00007fc4221226e0
R14:00007fc4287fcc10 R15:000000000000000f
RIP:0000000000489914 EFL:0000000000010202
CSGSFS:d7fb000000000033
22:M 10 May 01:18:59.744 # (00007fff173dcf1f) -> 000000000048b3ff
22:M 10 May 01:18:59.744 # (00007fff173dcf1e) -> 000000000000000f
22:M 10 May 01:18:59.744 # (00007fff173dcf1d) -> 0000000000000000
22:M 10 May 01:18:59.744 # (00007fff173dcf1c) -> 0000000000000008
22:M 10 May 01:18:59.744 # (00007fff173dcf1b) -> 0000000000000000
22:M 10 May 01:18:59.744 # (00007fff173dcf1a) -> 0000000000000000
22:M 10 May 01:18:59.744 # (00007fff173dcf19) -> 00007fc42a12ab40
22:M 10 May 01:18:59.744 # (00007fff173dcf18) -> 0000000000000a00
22:M 10 May 01:18:59.744 # (00007fff173dcf17) -> 0000000000000000
22:M 10 May 01:18:59.744 # (00007fff173dcf16) -> 000000000000a000
22:M 10 May 01:18:59.744 # (00007fff173dcf15) -> 00007fc42a200180
22:M 10 May 01:18:59.744 # (00007fff173dcf14) -> 0000000000000000
22:M 10 May 01:18:59.745 # (00007fff173dcf13) -> 0000000000000002
22:M 10 May 01:18:59.745 # (00007fff173dcf12) -> 000000010000000d
22:M 10 May 01:18:59.745 # (00007fff173dcf11) -> 00007fc4285ed941
22:M 10 May 01:18:59.745 # (00007fff173dcf10) -> 000000000000000f

------ FAST MEMORY TEST ------
22:M 10 May 01:18:59.745 # Bio thread for job type #0 terminated
22:M 10 May 01:18:59.745 # Bio thread for job type #1 terminated
22:M 10 May 01:18:59.745 # Bio thread for job type #2 terminated
*** Preparing to test memory region 736000 (94208 bytes)
*** Preparing to test memory region 2048000 (282624 bytes)
*** Preparing to test memory region 7fc422000000 (100663296 bytes)
*** Preparing to test memory region 7fc4285e0000 (16384 bytes)
*** Preparing to test memory region 7fc4287fe000 (8388608 bytes)
*** Preparing to test memory region 7fc428fff000 (8388608 bytes)
*** Preparing to test memory region 7fc429800000 (12582912 bytes)
*** Preparing to test memory region 7fc42a852000 (16384 bytes)
*** Preparing to test memory region 7fc42aa71000 (16384 bytes)
*** Preparing to test memory region 7fc42b233000 (16384 bytes)
*** Preparing to test memory region 7fc42b240000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: RM_StringToLongLong (base: 0x489910)
Module: /usr/local/lib/python2.7/dist-packages/ray/core/src/common/thirdparty/redis/src/redis-server *:6379 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x489910 -D -b binary -m i386:x86-64 /tmp/dump.bin

22:M 10 May 01:19:01.181 # dump of function (hexdump of 132 bytes):
4883ec08488b7f084889f20fb64fff89c883e0073c04775e0fb6c0ff24c5c8f24f00660f1f4400008b77f7e8300bfbff85c00f94c04883c4080fb6c0c30f1f00488b77efebe5662e0f1f840000000000c0e9030fb6f1ebd30f1f8400000000000fb677fdebc5662e0f1f8400000000000fb777fbebb531f6ebb1660f1f440000554889f5
Function at 0x43a470 is string2ll

=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@concretevitamin
Copy link
Contributor

concretevitamin commented May 10, 2018 via email

@robertnishihara
Copy link
Collaborator

Can you check if memory usage is steadily increasing as you run the job?

If it's an out-of-memory error, you can try addressing it by flushing keys periodically from Redis with

ray.experimental.flush_redis_unsafe()
ray.experimental.flush_task_and_object_metadata_unsafe()

@nikhilkhanna
Copy link
Author

@robertnishihara That seems to have fixed it the issue. Thanks a bunch!

@richardliaw richardliaw mentioned this issue Sep 12, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@robertnishihara @concretevitamin @nikhilkhanna