Redis crash due to corrupted data during aof rewrite

[MariaDima]

Hello,

We have a system with 15 redis nodes.
Each node runs 1 local redis instance and 3 cluster ones.
We faced a crash during the system start for node08 (Guru Meditation: "Unknown string encoding" #aof.c:781).

All nodes were started nearly at the same time.
From what I understand, a sync to master node07 started.
An AOF rewrite was triggered.
In the meanwhile the connection was lost since the connection was "moved" to node09.
Even though it's reported that aof rewrite was completed succesfully (at least from the server's perspective), it seems that the rewrite never occurred.
If we check the 012rdb_last_save_time, the last rewrite was on Friday, June 8, 2018 4:09:19 (when the server was started), while the triggered rewrite by node08 sync was on Friday, June 8, 2018 4:09:23.

For nearly 20-30 min, the connection to node09 could not be established.
We get the error "Error registering fd event for the new client: No such file or directory (fd=107).", but I'm not sure whether this relates to our crash.
We had another process running a redis client to this server.
This app crashed as well, so this could be a leftover from this connection and has nothing to do with the reported issue.
We see that in the rewrite action fd=107 is reported with an unknown addr (leftover from the node07 connection or leftover from the other client?).
When connection was finally established, we got ourselves into a recurring crash, due to Unknown string encoding.

The system got back to normal when we forced a server shutdown.

Our redis log:

2018-06-08T16:09:19.040998+00:00 local0.notice redis[13870]: Increased maximum number of open files to 10032 (it was originally set to 4096).
2018-06-08T16:09:19.042179+00:00 local0.notice redis[13870]: Redis 3.2.8 (00000000/0) 64 bit, standalone mode, port 6379, pid 13870 ready to start.
2018-06-08T16:09:19.042284+00:00 local0.warning redis[13870]: WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2018-06-08T16:09:19.042343+00:00 local0.warning redis[13870]: Server started, Redis version 3.2.8
2018-06-08T16:09:19.042410+00:00 local0.warning redis[13870]: WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2018-06-08T16:09:19.042502+00:00 local0.warning redis[13870]: WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2018-06-08T16:09:19.042570+00:00 local0.notice redis[13870]: The server is now ready to nodeept connections on port 6379
2018-06-08T16:09:19.138226+00:00 local0.notice redis[14000]: Increased maximum number of open files to 10032 (it was originally set to 4096).
2018-06-08T16:09:19.139389+00:00 local0.notice redis[14000]: Node configuration loaded, I'm node08b6379000000000000000000000000000000
2018-06-08T16:09:19.140135+00:00 local0.notice redis[14000]: Redis 3.2.8 (00000000/0) 64 bit, cluster mode, port 6379, pid 14000 ready to start.
2018-06-08T16:09:19.140235+00:00 local0.warning redis[14000]: WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2018-06-08T16:09:19.140305+00:00 local0.warning redis[14000]: Server started, Redis version 3.2.8
2018-06-08T16:09:19.140370+00:00 local0.warning redis[14000]: WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2018-06-08T16:09:19.140446+00:00 local0.warning redis[14000]: WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2018-06-08T16:09:19.249731+00:00 local0.notice redis[14089]: Increased maximum number of open files to 10032 (it was originally set to 4096).
2018-06-08T16:09:19.250845+00:00 local0.notice redis[14089]: Node configuration loaded, I'm node08b6380000000000000000000000000000000
2018-06-08T16:09:19.251579+00:00 local0.notice redis[14089]: Redis 3.2.8 (00000000/0) 64 bit, cluster mode, port 6380, pid 14089 ready to start.
2018-06-08T16:09:19.251665+00:00 local0.warning redis[14089]: WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2018-06-08T16:09:19.251719+00:00 local0.warning redis[14089]: Server started, Redis version 3.2.8
2018-06-08T16:09:19.251783+00:00 local0.warning redis[14089]: WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2018-06-08T16:09:19.251862+00:00 local0.warning redis[14089]: WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2018-06-08T16:09:19.425933+00:00 local0.notice redis[14271]: Increased maximum number of open files to 10032 (it was originally set to 4096).
2018-06-08T16:09:19.427273+00:00 local0.notice redis[14271]: Node configuration loaded, I'm node08b6381000000000000000000000000000000
2018-06-08T16:09:19.427942+00:00 local0.notice redis[14271]: Redis 3.2.8 (00000000/0) 64 bit, cluster mode, port 6381, pid 14271 ready to start.
2018-06-08T16:09:19.428023+00:00 local0.warning redis[14271]: WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
2018-06-08T16:09:19.428087+00:00 local0.warning redis[14271]: Server started, Redis version 3.2.8
2018-06-08T16:09:19.428161+00:00 local0.warning redis[14271]: WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2018-06-08T16:09:19.428235+00:00 local0.warning redis[14271]: WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
2018-06-08T16:09:19.941585+00:00 local0.notice redis[14089]: DB loaded from append only file: 0.690 seconds
2018-06-08T16:09:19.941674+00:00 local0.notice redis[14089]: The server is now ready to nodeept connections on port 6380
2018-06-08T16:09:19.949449+00:00 local0.warning redis[14089]: Cluster state changed: ok
2018-06-08T16:09:20.195559+00:00 local0.notice redis[14000]: DB loaded from append only file: 1.055 seconds
2018-06-08T16:09:20.196399+00:00 local0.notice redis[14000]: The server is now ready to nodeept connections on port 6379
2018-06-08T16:09:20.200595+00:00 local0.warning redis[14000]: Cluster state changed: ok
2018-06-08T16:09:20.200980+00:00 local0.notice redis[14000]: FAIL message received from node10b6379000000000000000000000000000000 about node08b6380000000000000000000000000000000
2018-06-08T16:09:20.211597+00:00 local0.notice redis[14000]: Clear FAIL state for node node08b6380000000000000000000000000000000: slave is reachable again.
2018-06-08T16:09:20.593644+00:00 local0.notice redis[14000]: FAIL message received from node12b6379000000000000000000000000000000 about node08b6381000000000000000000000000000000
2018-06-08T16:09:20.594811+00:00 local0.notice redis[14089]: FAIL message received from node12b6379000000000000000000000000000000 about node08b6381000000000000000000000000000000
2018-06-08T16:09:20.952916+00:00 local0.notice redis[14089]: Connecting to MASTER 139.142.213.196:6381
2018-06-08T16:09:20.953128+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync started
2018-06-08T16:09:20.954128+00:00 local0.notice redis[14089]: Non blocking connect for SYNC fired the event.
2018-06-08T16:09:20.955070+00:00 local0.notice redis[14089]: Master replied to PING, replication can continue...
2018-06-08T16:09:20.958239+00:00 local0.notice redis[14089]: Partial resynchronization not possible (no cached master)
2018-06-08T16:09:20.961936+00:00 local0.notice redis[14089]: Full resync from master: 391d60d908146440be4f59ae5afd2261bc5b7e18:1
2018-06-08T16:09:21.205583+00:00 local0.notice redis[14000]: Connecting to MASTER 139.142.213.195:6381
2018-06-08T16:09:21.205612+00:00 local0.notice redis[14000]: MASTER <-> SLAVE sync started
2018-06-08T16:09:21.205891+00:00 local0.notice redis[14000]: Non blocking connect for SYNC fired the event.
2018-06-08T16:09:21.206179+00:00 local0.notice redis[14000]: Master replied to PING, replication can continue...
2018-06-08T16:09:21.206639+00:00 local0.notice redis[14000]: Partial resynchronization not possible (no cached master)
2018-06-08T16:09:21.212700+00:00 local0.notice redis[14000]: Full resync from master: f08eff152593164b8795a2a2b17ec427e435ba39:1
2018-06-08T16:09:21.583293+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: receiving 25850978 bytes from master
2018-06-08T16:09:21.740672+00:00 local0.notice redis[14000]: MASTER <-> SLAVE sync: receiving 25650549 bytes from master
2018-06-08T16:09:21.896730+00:00 local0.notice redis[14271]: DB loaded from append only file: 2.468 seconds
2018-06-08T16:09:21.896811+00:00 local0.notice redis[14271]: The server is now ready to nodeept connections on port 6381
2018-06-08T16:09:21.901628+00:00 local0.warning redis[14271]: Cluster state changed: ok
2018-06-08T16:09:21.913962+00:00 local0.notice redis[14089]: Clear FAIL state for node node08b6381000000000000000000000000000000: slave is reachable again.
2018-06-08T16:09:21.913983+00:00 local0.notice redis[14000]: Clear FAIL state for node node08b6381000000000000000000000000000000: slave is reachable again.
2018-06-08T16:09:22.289173+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Flushing old data
2018-06-08T16:09:22.375142+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Loading DB in memory
2018-06-08T16:09:22.399136+00:00 local0.warning redis[14089]: Configuration change detected. Reconfiguring myself as a replica of node09b6379000000000000000000000000000000
2018-06-08T16:09:22.452604+00:00 local0.notice redis[14000]: MASTER <-> SLAVE sync: Flushing old data
2018-06-08T16:09:22.534214+00:00 local0.notice redis[14000]: MASTER <-> SLAVE sync: Loading DB in memory
2018-06-08T16:09:22.795382+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Finished with success
2018-06-08T16:09:22.798717+00:00 local0.notice redis[14089]: Background append only file rewriting started by pid 16182
2018-06-08T16:09:22.902873+00:00 local0.notice redis[14271]: Connecting to MASTER 139.142.213.199:6379
2018-06-08T16:09:22.903099+00:00 local0.notice redis[14271]: MASTER <-> SLAVE sync started
2018-06-08T16:09:22.903476+00:00 local0.notice redis[14271]: Non blocking connect for SYNC fired the event.
2018-06-08T16:09:22.905249+00:00 local0.notice redis[14271]: Master replied to PING, replication can continue...
2018-06-08T16:09:22.905813+00:00 local0.notice redis[14271]: Partial resynchronization not possible (no cached master)
2018-06-08T16:09:22.909128+00:00 local0.notice redis[14271]: Full resync from master: 57f0e588c128037f8d8884bae2e003055c6fd023:1
2018-06-08T16:09:22.958210+00:00 local0.notice redis[14000]: MASTER <-> SLAVE sync: Finished with success
2018-06-08T16:09:22.960012+00:00 local0.notice redis[14000]: Background append only file rewriting started by pid 16203
2018-06-08T16:09:23.318240+00:00 local0.notice redis[14089]: AOF rewrite child asks to stop sending diffs.
2018-06-08T16:09:23.318364+00:00 local0.notice redis[16182]: Parent agreed to stop sending diffs. Finalizing AOF...
2018-06-08T16:09:23.319633+00:00 local0.notice redis[16182]: Concatenating 0.00 MB of AOF diff received from parent.
2018-06-08T16:09:23.319830+00:00 local0.notice redis[16182]: SYNC append only file rewrite performed
2018-06-08T16:09:23.320799+00:00 local0.notice redis[16182]: AOF rewrite: 6 MB of memory used by copy-on-write
2018-06-08T16:09:23.400751+00:00 local0.notice redis[14089]: Background AOF rewrite terminated with success
2018-06-08T16:09:23.400780+00:00 local0.notice redis[14089]: Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
2018-06-08T16:09:23.404454+00:00 local0.notice redis[14089]: Background AOF rewrite finished successfully
2018-06-08T16:09:23.414299+00:00 local0.notice redis[14000]: AOF rewrite child asks to stop sending diffs.
2018-06-08T16:09:23.414436+00:00 local0.notice redis[16203]: Parent agreed to stop sending diffs. Finalizing AOF...
2018-06-08T16:09:23.414524+00:00 local0.notice redis[16203]: Concatenating 0.00 MB of AOF diff received from parent.
2018-06-08T16:09:23.414628+00:00 local0.notice redis[16203]: SYNC append only file rewrite performed
2018-06-08T16:09:23.415662+00:00 local0.notice redis[16203]: AOF rewrite: 4 MB of memory used by copy-on-write
2018-06-08T16:09:23.473615+00:00 local0.notice redis[14000]: Background AOF rewrite terminated with success
2018-06-08T16:09:23.473644+00:00 local0.notice redis[14000]: Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
2018-06-08T16:09:23.473649+00:00 local0.notice redis[14000]: Background AOF rewrite finished successfully
2018-06-08T16:09:23.502927+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:23.551787+00:00 local0.warning redis[14089]: last message repeated 2 times
2018-06-08T16:09:23.551688+00:00 local0.notice redis[14271]: MASTER <-> SLAVE sync: receiving 27550834 bytes from master
2018-06-08T16:09:23.887954+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:24.077543+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:24.330448+00:00 local0.notice redis[14271]: MASTER <-> SLAVE sync: Flushing old data
2018-06-08T16:09:24.351524+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:24.510511+00:00 local0.notice redis[14271]: MASTER <-> SLAVE sync: Loading DB in memory
2018-06-08T16:09:24.526753+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:24.971807+00:00 local0.warning redis[14089]: last message repeated 4 times
2018-06-08T16:09:24.971745+00:00 local0.notice redis[14271]: MASTER <-> SLAVE sync: Finished with success
2018-06-08T16:09:24.979001+00:00 local0.notice redis[14271]: Background append only file rewriting started by pid 16899
2018-06-08T16:09:25.223233+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:25.302874+00:00 local0.notice redis[14271]: AOF rewrite child asks to stop sending diffs.
2018-06-08T16:09:25.302984+00:00 local0.notice redis[16899]: Parent agreed to stop sending diffs. Finalizing AOF...
2018-06-08T16:09:25.303074+00:00 local0.notice redis[16899]: Concatenating 0.00 MB of AOF diff received from parent.
2018-06-08T16:09:25.303363+00:00 local0.notice redis[16899]: SYNC append only file rewrite performed
2018-06-08T16:09:25.304923+00:00 local0.notice redis[16899]: AOF rewrite: 5 MB of memory used by copy-on-write
2018-06-08T16:09:25.382098+00:00 local0.notice redis[14271]: Background AOF rewrite terminated with success
2018-06-08T16:09:25.382146+00:00 local0.notice redis[14271]: Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
2018-06-08T16:09:25.382276+00:00 local0.notice redis[14271]: Background AOF rewrite finished successfully
2018-06-08T16:09:26.404887+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:09:57.522279+00:00 local0.warning redis[14089]: last message repeated 32 times
2018-06-08T16:10:23.543121+00:00 local0.warning redis[14089]: last message repeated 15 times
2018-06-08T16:10:23.542897+00:00 local0.warning redis[14089]: MASTER timeout: no data nor PING received...
2018-06-08T16:10:23.542989+00:00 local0.warning redis[14089]: Connection with master lost.
2018-06-08T16:10:23.543041+00:00 local0.notice redis[14089]: Caching the disconnected master state.
2018-06-08T16:10:23.543090+00:00 local0.notice redis[14089]: Connecting to MASTER 139.142.213.198:6379
2018-06-08T16:10:23.543298+00:00 local0.warning redis[14089]: Can't create readable event for SYNC
2018-06-08T16:10:23.620012+00:00 local0.warning redis[14089]: Error registering fd event for the new client: No such file or directory (fd=107)
2018-06-08T16:10:24.545561+00:00 local0.warning redis[14089]: last message repeated 3 times
2018-06-08T16:10:24.545373+00:00 local0.notice redis[14089]: Connecting to MASTER 139.142.213.198:6379
2018-06-08T16:10:24.545421+00:00 local0.warning redis[14089]: Can't create readable event for SYNC
...
2018-06-08T16:35:29.228948+00:00 local0.notice redis[14089]: Connecting to MASTER 139.142.213.198:6379
2018-06-08T16:35:29.228985+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync started
2018-06-08T16:35:29.229581+00:00 local0.notice redis[14089]: Non blocking connect for SYNC fired the event.
2018-06-08T16:35:29.230008+00:00 local0.notice redis[14089]: Master replied to PING, replication can continue...
2018-06-08T16:35:29.230702+00:00 local0.notice redis[14089]: Trying a partial resynchronization (request 391d60d908146440be4f59ae5afd2261bc5b7e18:2).
2018-06-08T16:35:29.233903+00:00 local0.notice redis[14089]: Full resync from master: ebb7a228b02e6b0d8fdc656e160eeae5dd716253:51554
2018-06-08T16:35:29.233925+00:00 local0.notice redis[14089]: Discarding previously cached master state.
2018-06-08T16:35:29.874268+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: receiving 25850762 bytes from master
2018-06-08T16:35:30.631844+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Flushing old data
2018-06-08T16:35:30.727835+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Loading DB in memory
2018-06-08T16:35:31.105659+00:00 local0.notice redis[14089]: MASTER <-> SLAVE sync: Finished with success
2018-06-08T16:35:31.107469+00:00 local0.notice redis[14089]: Background append only file rewriting started by pid 7179
2018-06-08T16:35:31.149525+00:00 local0.warning redis[7179]: #12#012=== REDIS BUG REPORT START: Cut & paste starting from here ===
2018-06-08T16:35:31.149633+00:00 local0.warning redis[7179]: ------------------------------------------------
2018-06-08T16:35:31.149693+00:00 local0.warning redis[7179]: !!! Software Failure. Press left mouse button to continue
2018-06-08T16:35:31.149747+00:00 local0.warning redis[7179]: Guru Meditation: "Unknown string encoding" #aof.c:781
2018-06-08T16:35:31.149799+00:00 local0.warning redis[7179]: (forcing SIGSEGV in order to print the stack trace)
2018-06-08T16:35:31.149849+00:00 local0.warning redis[7179]: ------------------------------------------------
2018-06-08T16:35:31.149912+00:00 local0.warning redis[7179]: Redis 3.2.8 crashed by signal: 11
2018-06-08T16:35:31.149964+00:00 local0.warning redis[7179]: Crashed running the instuction at: 0x46186e
2018-06-08T16:35:31.150014+00:00 local0.warning redis[7179]: nodeessing address: 0xffffffffffffffff
2018-06-08T16:35:31.150066+00:00 local0.warning redis[7179]: Failed assertion: (:0)
2018-06-08T16:35:31.150110+00:00 local0.warning redis[7179]: #12------ STACK TRACE ------
2018-06-08T16:35:31.150819+00:00 local0.warning redis[7179]: #12------ INFO OUTPUT ------
2018-06-08T16:35:31.151087+00:00 local0.warning redis[7179]: # Server#015#012redis_version:3.2.8#015#012redis_git_sha1:00000000#015#012redis_git_dirty:0#015#012redis_build_id:dfd8c65c90e3a499#015#012redis_mode:cluster#015#012os:Linux 3.0.101-108.21-default x86_64#015#012arch_bits:64#015#012multiplexing_api:epoll#015#012gcc_version:4.3.4#015#012process_id:7179#015#012run_id:46e362d6e5e88bb2fc2442d87f0e37f154e2e301#015#012tcp_port:6380#015#012uptime_in_seconds:1572#015#012uptime_in_days:0#015#012hz:10#015#012lru_clock:1749074#015#012executable:/usr/sbin/redis-cluster#015#012config_file:/etc/redis/redis-cluster.conf#015#012#015#012# Clients#015#012connected_clients:24#015#012client_longest_output_list:0#015#012client_biggest_input_buf:0#015#012blocked_clients:0#015#012#015#012# Memory#015#012used_memory:43372664#015#012used_memory_human:41.36M#015#012used_memory_rss:52629504#015#012used_memory_rss_human:50.19M#015#012used_memory_peak:43667656#015#012used_memory_peak_human:41.64M#015#012total_system_memory:8255938560#015#012total_system_memory_human:7.69G#015#012used_memory_lua:37888#015#012used_memory_lua_human:37.00K#015#012maxmemory:0#015#012maxmemory_human:0B#015#012maxmemory_policy:noeviction#015#012mem_fragmentation_ratio:1.21#015#012mem_allocator:jemalloc-4.0.3#015#012#015#012# Persistence#015#012loading:0#015#012rdb_changes_since_last_save:116050#15#012rdb_bgsave_in_progress:0#015#012rdb_last_save_time:1528474159#015#012rdb_last_bgsave_status:ok#015#012rdb_last_bgsave_time_sec:-1#015#012rdb_current_bgsave_time_sec:-1#015#012aof_enabled:0#015#012aof_rewrite_in_progress:0#015#012aof_rewrite_scheduled:0#015#012aof_last_rewrite_time_sec:1#015#012aof_current_rewrite_time_sec:-1#015#012aof_last_bgrewrite_status:ok#015#012aof_last_write_status:ok#015#012#015#012# Stats#015#012total_connections_received:46#015#012total_commands_processed:128#015#012instantaneous_ops_per_sec:4#015#012total_net_input_bytes:51704729#015#012total_net_output_bytes:491038#015#012instantaneous_input_kbps:14579.10#015#012instantaneous_output_kbps:17.68#015#012reje
2018-06-08T16:35:31.151161+00:00 local0.warning redis[7179]: #12------ CLIENT LIST OUTPUT ------
2018-06-08T16:35:31.151440+00:00 local0.warning redis[7179]: id=17 addr=139.142.213.199:60237 fd=113 name= age=1570 idle=1570 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=12 addr=139.142.213.199:60173 fd=108 name= age=1571 idle=1571 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=13 addr=139.142.213.194:49798 fd=109 name= age=1570 idle=1570 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=48 addr=139.142.213.202:47937 fd=117 name= age=2 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=cluster#012id=14 addr=139.142.213.193:50195 fd=110 name= age=1570 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=cluster#012id=7 addr=139.142.213.199:59991 fd=101 name= age=1571 idle=1571 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=8 addr=139.142.213.199:60060 fd=102 name= age=1571 idle=853 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=cluster#012id=30 addr=139.142.213.196:63575 fd=6 name= age=1568 idle=1568 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=19 addr=139.142.213.199:60290 fd=115 name= age=1570 idle=71 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=cluster#012id=3 addr=139.142.213.190:64051 fd=11 name= age=1572 idle=1393 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=cluster#012id=24 addr=139.142.213.192:49130 fd=121 name= age=1569 idle=1568 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=27 addr=139.142.213.204:59547 fd=106 name= age=1569 idle=1568 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info#012id=28 addr=?:0 fd=107 name= age=1569 idle=1569 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0
2018-06-08T16:35:31.151507+00:00 local0.warning redis[7179]: #12------ REGISTERS ------
2018-06-08T16:35:31.151567+00:00 local0.warning redis[7179]: #012RAX:0000000000000000 RBX:000000000000030d#012RCX:0000000000000000 RDX:0000000000000060#012RDI:00007fb5956b3e80 RSI:00007fb59547e5e0#012RBP:0000000000000000 RSP:00007ffc84a4ed90#012R8 :0000000000000000 R9 :0000000000000000#012R10:0000000000004000 R11:0000000000000000#012R12:00000000004eb703 R13:00000000004e6168#012R14:00007fb5915b9460 R15:00007ffc84a4ee00#012RIP:000000000046186e EFL:0000000000010206#012CSGSFS:2d2d000000000033
2018-06-08T16:35:31.151623+00:00 local0.warning redis[7179]: (00007ffc84a4ed9f) -> 0000000000475000
2018-06-08T16:35:31.151673+00:00 local0.warning redis[7179]: (00007ffc84a4ed9e) -> 0000000000475090
2018-06-08T16:35:31.151723+00:00 local0.warning redis[7179]: (00007ffc84a4ed9d) -> 0000000000000000
2018-06-08T16:35:31.151776+00:00 local0.warning redis[7179]: (00007ffc84a4ed9c) -> 00007fb594e27800
2018-06-08T16:35:31.151825+00:00 local0.warning redis[7179]: (00007ffc84a4ed9b) -> 0000000000a366b2
2018-06-08T16:35:31.151874+00:00 local0.warning redis[7179]: (00007ffc84a4ed9a) -> 00000163e040c4a3
2018-06-08T16:35:31.151927+00:00 local0.warning redis[7179]: (00007ffc84a4ed99) -> 00000000007415c0
2018-06-08T16:35:31.151977+00:00 local0.warning redis[7179]: (00007ffc84a4ed98) -> 00007fb5916a9790
2018-06-08T16:35:31.152027+00:00 local0.warning redis[7179]: (00007ffc84a4ed97) -> 00007ffc84a4f010
2018-06-08T16:35:31.152077+00:00 local0.warning redis[7179]: (00007ffc84a4ed96) -> 00000000fbad8001
2018-06-08T16:35:31.152127+00:00 local0.warning redis[7179]: (00007ffc84a4ed95) -> 000000000045ebaa
2018-06-08T16:35:31.152176+00:00 local0.warning redis[7179]: (00007ffc84a4ed94) -> 00007ffc84a4ef90
2018-06-08T16:35:31.152225+00:00 local0.warning redis[7179]: (00007ffc84a4ed93) -> 000000000045de7e
2018-06-08T16:35:31.152282+00:00 local0.warning redis[7179]: (00007ffc84a4ed92) -> ffffffffffffffff
2018-06-08T16:35:31.152331+00:00 local0.warning redis[7179]: (00007ffc84a4ed91) -> 00007ffc84a4ef90
2018-06-08T16:35:31.152381+00:00 local0.warning redis[7179]: (00007ffc84a4ed90) -> 000000000000000d
2018-06-08T16:35:31.152426+00:00 local0.warning redis[7179]: #12------ FAST MEMORY TEST ------
2018-06-08T16:35:31.424991+00:00 local0.warning redis[7179]: Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
2018-06-08T16:35:31.425113+00:00 local0.warning redis[7179]: #12------ DUMPING CODE AROUND EIP ------#012Symbol: _serverPanic (base: 0x4617f0)#012Module: redis-aof-rewrite 139.142.213.197:6380 [cluster] (base 0x400000)#12$ xxd -r -p /tmp/dump.hex /tmp/dump.bin#012$ objdump --adjust-vma=0x4617f0 -D -b binary -m i386:x86-64 /tmp/dump.bin#012------
2018-06-08T16:35:31.425176+00:00 local0.warning redis[7179]: dump of function (hexdump of 254 bytes):
2018-06-08T16:35:31.425219+00:00 local0.warning redis[7179]: 48895c24e84c896424f089d34c896c24f84883ec184989fd4989f4e8b0ffffff
2018-06-08T16:35:31.425257+00:00 local0.warning redis[7179]: bea8c04e00bf0300000031c0e87f90fcffbee0c04e00bf0300000031c0e86e90
2018-06-08T16:35:31.425295+00:00 local0.warning redis[7179]: fcff4189d84c89e14c89eabee8ba4e00bf0300000031c0e85490fcffbe20c14e
2018-06-08T16:35:31.425332+00:00 local0.warning redis[7179]: 00bf0300000031c0e84390fcffbea8c04e00bf0300000031c0e83290fcffc604
2018-06-08T16:35:31.425370+00:00 local0.warning redis[7179]: 25ffffffff78488b1c244c8b6424084c8b6c24104883c418c30f1f8000000000
2018-06-08T16:35:31.425407+00:00 local0.warning redis[7179]: 48895c24e84c896424f04889fb4c896c24f84883ec184189d54989f4e80fffff
2018-06-08T16:35:31.425444+00:00 local0.warning redis[7179]: ffbe03bb4e00bf0300000031c0e8de8ffcff4989d84489e94c89e2be1cbb4e00
2018-06-08T16:35:31.425482+00:00 local0.warning redis[7179]: bf0300000031c0e8c48ffcffbe58c14e00bf0300000031c048891d09df2b
2018-06-08T16:35:31.425520+00:00 local0.warning redis[7179]:
2018-06-08T16:35:31.425727+00:00 local0.warning redis[7179]: #12=== REDIS BUG REPORT END. Make sure to include from START to END. ===#12#012 Please report the crash by opening an issue on github:#12#012 http://github.com/antirez/redis/issues#012#012 Suspect RAM error? Use redis-server --test-memory to verify it.#12
2018-06-08T16:35:31.507627+00:00 local0.warning redis[14089]: Background AOF rewrite terminated by signal 11