Redis-Sentinel segfault on malformed command #1541

Closed
kpthill opened this Issue Feb 7, 2014 · 2 comments

Comments

Projects
None yet
2 participants

kpthill commented Feb 7, 2014

I had redis sentinel running against a couple of machines and typo'd the sentinel masters command as sentinel master. This caused a segfault. Upon investigation, I found these steps sufficient to reproduce the error:

  1. redis-sentinel sentinel_base.conf --port 26379. You can have a master running to monitor or not.
  2. redis-cli -p 26379 sentinel master

This gives Error: Server closed the connection and the redis-sentinel stdout is at the bottom of this bug.

The machine is a Retina MBP, OSX 10.9.1.

redis-server --version gives Redis server v=2.8.4 sha=00000000:0 malloc=libc bits=64 build=92637893332b8579.

sentinel_base.conf:

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 3000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 6000

Sentinel error messages:

[78943] 06 Feb 17:48:26.107 * Max number of open files set to 10032
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 2.8.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 78943
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

[78943] 06 Feb 17:48:26.109 # Sentinel runid is f7cacfea5f82f221eda8ec368115539ae5745d03
[78943] 06 Feb 17:48:27.902 # 

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[78943] 06 Feb 17:48:27.902 #     Redis 2.8.4 crashed by signal: 11
[78943] 06 Feb 17:48:27.902 #     Failed assertion: <no assertion failed> (<no file>:0)
[78943] 06 Feb 17:48:27.902 # --- STACK TRACE
0   redis-sentinel                      0x0000000107a9cb43 logStackTrace + 103
1   redis-sentinel                      0x0000000107aa7980 sentinelGetMasterByNameOrReplyError + 28
2   libsystem_platform.dylib            0x00007fff9230f5aa _sigtramp + 26
3   ???                                 0x0000000000000000 0x0 + 0
4   redis-sentinel                      0x0000000107aa3740 sentinelCommand + 368
5   redis-sentinel                      0x0000000107a75187 call + 197
6   redis-sentinel                      0x0000000107a75674 processCommand + 854
7   redis-sentinel                      0x0000000107a7e491 processInputBuffer + 110
8   redis-sentinel                      0x0000000107a7c761 readQueryFromClient + 415
9   redis-sentinel                      0x0000000107a70285 aeProcessEvents + 500
10  redis-sentinel                      0x0000000107a7046c aeMain + 43
11  redis-sentinel                      0x0000000107a774e2 main + 832
12  libdyld.dylib                       0x00007fff8f8bc5fd start + 1
13  ???                                 0x0000000000000002 0x0 + 2
[78943] 06 Feb 17:48:27.902 # --- INFO OUTPUT
[78943] 06 Feb 17:48:27.902 # # Server
redis_version:2.8.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:92637893332b8579
redis_mode:sentinel
os:Darwin 13.0.0 x86_64
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:78943
run_id:f7cacfea5f82f221eda8ec368115539ae5745d03
tcp_port:26379
uptime_in_seconds:1
uptime_in_days:0
hz:15
lru_clock:761738
config_file:/Users/pthill/Work/sentinel_testing/sentinel_base.conf

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:993584
used_memory_human:970.30K
used_memory_rss:1777664
used_memory_peak:942112
used_memory_peak_human:920.03K
used_memory_lua:33792
mem_fragmentation_ratio:1.79
mem_allocator:libc

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1391737706
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:1
total_commands_processed:0
instantaneous_ops_per_sec:0
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.01
used_cpu_user:0.01
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats

# Keyspace
hash_init_value: 1391756969

[78943] 06 Feb 17:48:27.902 # --- CLIENT LIST OUTPUT
[78943] 06 Feb 17:48:27.902 # addr=127.0.0.1:56710 fd=6 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sentinel

[78943] 06 Feb 17:48:27.902 # --- CURRENT CLIENT INFO
[78943] 06 Feb 17:48:27.902 # client: addr=127.0.0.1:56710 fd=6 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sentinel
[78943] 06 Feb 17:48:27.902 # argv[0]: 'sentinel'
[78943] 06 Feb 17:48:27.902 # argv[1]: 'master'
[78943] 06 Feb 17:48:27.902 # --- REGISTERS
[78943] 06 Feb 17:48:27.902 # 
RAX:0000000000000006 RBX:00007f9f6c800000
RCX:00000000000003e8 RDX:00007fff7d9cf920
RDI:00007f9f69c04a60 RSI:0000000107ac7c44
RBP:00007fff581915a0 RSP:00007fff58191590
R8 :0000000052f43b6b R9 :000000000000007c
R10:002cb2cbacb2eb2d R11:0000000000000002
R12:00007f9f6c800000 R13:0000000000000000
R14:00007f9f69c04228 R15:00007f9f6c800000
RIP:0000000107aa7980 EFL:0000000000010246
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff5819159f) -> 0000000000000007
[78943] 06 Feb 17:48:27.902 # (00007fff5819159e) -> cd001df30b1873d0
[78943] 06 Feb 17:48:27.902 # (00007fff5819159d) -> 00007fff962dc273
[78943] 06 Feb 17:48:27.902 # (00007fff5819159c) -> 00007fff58191630
[78943] 06 Feb 17:48:27.902 # (00007fff5819159b) -> 00007f9f6c800000
[78943] 06 Feb 17:48:27.902 # (00007fff5819159a) -> 0000000107ad90f8
[78943] 06 Feb 17:48:27.902 # (00007fff58191599) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff58191598) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff58191597) -> 00007fff94c8dbd9
[78943] 06 Feb 17:48:27.902 # (00007fff58191596) -> 00007fff581915d0
[78943] 06 Feb 17:48:27.902 # (00007fff58191595) -> 00007f9f00000006
[78943] 06 Feb 17:48:27.902 # (00007fff58191594) -> 00007f9f69c00000
[78943] 06 Feb 17:48:27.902 # (00007fff58191593) -> 0000000107aa3740
[78943] 06 Feb 17:48:27.902 # (00007fff58191592) -> 00007fff58191630
[78943] 06 Feb 17:48:27.902 # (00007fff58191591) -> 00007f9f69c04a40
[78943] 06 Feb 17:48:27.902 # (00007fff58191590) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # 
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash opening an issue on github:

           http://github.com/antirez/redis/issues

  Suspect RAM error? Use redis-server --test-memory to veryfy it.


Segmentation fault: 11
Contributor

mattsta commented Feb 7, 2014

Thanks for the report! Looks like this was fixed on Jan 31:

commit 3ff1bb4b2e146f94672967f1a37bd01b83cd6150
Author: antirez <antirez@gmail.com>
Date:   Fri Jan 31 10:13:36 2014 +0100

    Sentinel: check arity for SENTINEL MASTER command.

    This fixes issue #1530.

I just tested your conditions and the server returns an error properly now instead of crashing.

You can get the fix with the recently released Redis 2.8.5 or you can grab a new version from git.

kpthill commented Feb 7, 2014

Great, thanks.

@kpthill kpthill closed this Feb 7, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment