Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Redis-Sentinel segfault on malformed command #1541

Closed
kpthill opened this Issue · 2 comments

2 participants

@kpthill

I had redis sentinel running against a couple of machines and typo'd the sentinel masters command as sentinel master. This caused a segfault. Upon investigation, I found these steps sufficient to reproduce the error:

1) redis-sentinel sentinel_base.conf --port 26379. You can have a master running to monitor or not.
2) redis-cli -p 26379 sentinel master

This gives Error: Server closed the connection and the redis-sentinel stdout is at the bottom of this bug.

The machine is a Retina MBP, OSX 10.9.1.

redis-server --version gives Redis server v=2.8.4 sha=00000000:0 malloc=libc bits=64 build=92637893332b8579.

sentinel_base.conf:

sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 3000
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster 6000

Sentinel error messages:

[78943] 06 Feb 17:48:26.107 * Max number of open files set to 10032
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 2.8.4 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                   
 (    '      ,       .-`  | `,    )     Running in sentinel mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 26379
 |    `-._   `._    /     _.-'    |     PID: 78943
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           http://redis.io        
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

[78943] 06 Feb 17:48:26.109 # Sentinel runid is f7cacfea5f82f221eda8ec368115539ae5745d03
[78943] 06 Feb 17:48:27.902 # 

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[78943] 06 Feb 17:48:27.902 #     Redis 2.8.4 crashed by signal: 11
[78943] 06 Feb 17:48:27.902 #     Failed assertion: <no assertion failed> (<no file>:0)
[78943] 06 Feb 17:48:27.902 # --- STACK TRACE
0   redis-sentinel                      0x0000000107a9cb43 logStackTrace + 103
1   redis-sentinel                      0x0000000107aa7980 sentinelGetMasterByNameOrReplyError + 28
2   libsystem_platform.dylib            0x00007fff9230f5aa _sigtramp + 26
3   ???                                 0x0000000000000000 0x0 + 0
4   redis-sentinel                      0x0000000107aa3740 sentinelCommand + 368
5   redis-sentinel                      0x0000000107a75187 call + 197
6   redis-sentinel                      0x0000000107a75674 processCommand + 854
7   redis-sentinel                      0x0000000107a7e491 processInputBuffer + 110
8   redis-sentinel                      0x0000000107a7c761 readQueryFromClient + 415
9   redis-sentinel                      0x0000000107a70285 aeProcessEvents + 500
10  redis-sentinel                      0x0000000107a7046c aeMain + 43
11  redis-sentinel                      0x0000000107a774e2 main + 832
12  libdyld.dylib                       0x00007fff8f8bc5fd start + 1
13  ???                                 0x0000000000000002 0x0 + 2
[78943] 06 Feb 17:48:27.902 # --- INFO OUTPUT
[78943] 06 Feb 17:48:27.902 # # Server
redis_version:2.8.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:92637893332b8579
redis_mode:sentinel
os:Darwin 13.0.0 x86_64
arch_bits:64
multiplexing_api:kqueue
gcc_version:4.2.1
process_id:78943
run_id:f7cacfea5f82f221eda8ec368115539ae5745d03
tcp_port:26379
uptime_in_seconds:1
uptime_in_days:0
hz:15
lru_clock:761738
config_file:/Users/pthill/Work/sentinel_testing/sentinel_base.conf

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:993584
used_memory_human:970.30K
used_memory_rss:1777664
used_memory_peak:942112
used_memory_peak_human:920.03K
used_memory_lua:33792
mem_fragmentation_ratio:1.79
mem_allocator:libc

# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1391737706
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

# Stats
total_connections_received:1
total_commands_processed:0
instantaneous_ops_per_sec:0
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.01
used_cpu_user:0.01
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats

# Keyspace
hash_init_value: 1391756969

[78943] 06 Feb 17:48:27.902 # --- CLIENT LIST OUTPUT
[78943] 06 Feb 17:48:27.902 # addr=127.0.0.1:56710 fd=6 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sentinel

[78943] 06 Feb 17:48:27.902 # --- CURRENT CLIENT INFO
[78943] 06 Feb 17:48:27.902 # client: addr=127.0.0.1:56710 fd=6 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sentinel
[78943] 06 Feb 17:48:27.902 # argv[0]: 'sentinel'
[78943] 06 Feb 17:48:27.902 # argv[1]: 'master'
[78943] 06 Feb 17:48:27.902 # --- REGISTERS
[78943] 06 Feb 17:48:27.902 # 
RAX:0000000000000006 RBX:00007f9f6c800000
RCX:00000000000003e8 RDX:00007fff7d9cf920
RDI:00007f9f69c04a60 RSI:0000000107ac7c44
RBP:00007fff581915a0 RSP:00007fff58191590
R8 :0000000052f43b6b R9 :000000000000007c
R10:002cb2cbacb2eb2d R11:0000000000000002
R12:00007f9f6c800000 R13:0000000000000000
R14:00007f9f69c04228 R15:00007f9f6c800000
RIP:0000000107aa7980 EFL:0000000000010246
CS :000000000000002b FS:0000000000000000  GS:0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff5819159f) -> 0000000000000007
[78943] 06 Feb 17:48:27.902 # (00007fff5819159e) -> cd001df30b1873d0
[78943] 06 Feb 17:48:27.902 # (00007fff5819159d) -> 00007fff962dc273
[78943] 06 Feb 17:48:27.902 # (00007fff5819159c) -> 00007fff58191630
[78943] 06 Feb 17:48:27.902 # (00007fff5819159b) -> 00007f9f6c800000
[78943] 06 Feb 17:48:27.902 # (00007fff5819159a) -> 0000000107ad90f8
[78943] 06 Feb 17:48:27.902 # (00007fff58191599) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff58191598) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # (00007fff58191597) -> 00007fff94c8dbd9
[78943] 06 Feb 17:48:27.902 # (00007fff58191596) -> 00007fff581915d0
[78943] 06 Feb 17:48:27.902 # (00007fff58191595) -> 00007f9f00000006
[78943] 06 Feb 17:48:27.902 # (00007fff58191594) -> 00007f9f69c00000
[78943] 06 Feb 17:48:27.902 # (00007fff58191593) -> 0000000107aa3740
[78943] 06 Feb 17:48:27.902 # (00007fff58191592) -> 00007fff58191630
[78943] 06 Feb 17:48:27.902 # (00007fff58191591) -> 00007f9f69c04a40
[78943] 06 Feb 17:48:27.902 # (00007fff58191590) -> 0000000000000000
[78943] 06 Feb 17:48:27.902 # 
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

       Please report the crash opening an issue on github:

           http://github.com/antirez/redis/issues

  Suspect RAM error? Use redis-server --test-memory to veryfy it.


Segmentation fault: 11
@mattsta

Thanks for the report! Looks like this was fixed on Jan 31:

commit 3ff1bb4b2e146f94672967f1a37bd01b83cd6150
Author: antirez <antirez@gmail.com>
Date:   Fri Jan 31 10:13:36 2014 +0100

    Sentinel: check arity for SENTINEL MASTER command.

    This fixes issue #1530.

I just tested your conditions and the server returns an error properly now instead of crashing.

You can get the fix with the recently released Redis 2.8.5 or you can grab a new version from git.

@kpthill

Great, thanks.

@kpthill kpthill closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.