Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Crash when ran out of memory #1524

Closed
noahd1 opened this Issue · 8 comments

3 participants

@noahd1

Recently our Redis server crashed when it hit the "maxmemory" limit. I understand that this is a very unhealthy place to be, but I also understand that a crash is not the expected behavior, but that instead it should follow the maxmemory policy in our config file (noeviction). Thoughts? Is there any other information I can provide?

    [15035] 23 Jan 21:22:42.807 # Out Of Memory allocating 6102005 bytes!
    [15035] 23 Jan 21:22:42.817 #

    === REDIS BUG REPORT START: Cut & paste starting from here ===
    [15035] 23 Jan 21:22:42.817 # ------------------------------------------------
    [15035] 23 Jan 21:22:42.818 # !!! Software Failure. Press left mouse button to continue
    [15035] 23 Jan 21:22:42.818 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:2914
    [15035] 23 Jan 21:22:42.818 # (forcing SIGSEGV in order to print the stack trace)
    [15035] 23 Jan 21:22:42.818 # ------------------------------------------------
    [15035] 23 Jan 21:22:42.818 #     Redis 2.8.2 crashed by signal: 11
    [15035] 23 Jan 21:22:42.818 #     Failed assertion: <no assertion failed> (<no file>:0)
    [15035] 23 Jan 21:22:42.818 # --- STACK TRACE
    /opt/redis/bin/redis-server 0.0.0.0:6381(logStackTrace+0x43)[0x445813]
    /opt/redis/bin/redis-server 0.0.0.0:6381(_redisPanic+0x7f)[0x44532f]
    /lib64/libpthread.so.0(+0xf710)[0x7f3b0213e710]
    /opt/redis/bin/redis-server 0.0.0.0:6381(_redisPanic+0x7f)[0x44532f]
    /opt/redis/bin/redis-server 0.0.0.0:6381(redisOutOfMemoryHandler+0x2c)[0x41cfbc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(zrealloc+0xf9)[0x41fda9]
    /opt/redis/bin/redis-server 0.0.0.0:6381(sdsMakeRoomFor+0x56)[0x41e7a6]
    /opt/redis/bin/redis-server 0.0.0.0:6381(processMultibulkBuffer+0x30c)[0x4261cc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(processInputBuffer+0x82)[0x426492]
    /opt/redis/bin/redis-server 0.0.0.0:6381(readQueryFromClient+0xb8)[0x426588]
    /opt/redis/bin/redis-server 0.0.0.0:6381(aeProcessEvents+0x13c)[0x4164dc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(aeMain+0x2b)[0x41679b]
    /opt/redis/bin/redis-server 0.0.0.0:6381(main+0x2bf)[0x41ddbf]
    /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f3b01db9d1d]
    /opt/redis/bin/redis-server 0.0.0.0:6381[0x415af9]
    [15035] 23 Jan 21:22:42.849 # --- INFO OUTPUT
    [15035] 23 Jan 21:22:42.850 # # Server
    redis_version:2.8.2
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:c820c8bf89592140
    redis_mode:standalone
    os:Linux 2.6.32-042stab079.5 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:15035
    run_id:b34ff247f5ff4ba0a792e146fe9efdd2cf271b56
    tcp_port:6381
    uptime_in_seconds:177185
    uptime_in_days:2
    hz:10
    lru_clock:639184
    config_file:/etc/redis/6381.conf

    # Clients
    connected_clients:48
    client_longest_output_list:0
    client_biggest_input_buf:1869295
    blocked_clients:0

    # Memory
    used_memory:6228251792
    used_memory_human:5.80G
    used_memory_rss:4386205696
    used_memory_peak:6461639456
    used_memory_peak_human:6.02G
    used_memory_lua:48128
    mem_fragmentation_ratio:0.70
    mem_allocator:jemalloc-3.2.0

    # Persistence
    loading:0
    rdb_changes_since_last_save:1051239090
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1390335001
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:1
    rdb_current_bgsave_time_sec:-1
    aof_enabled:0
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:-1
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok

    # Stats
    total_connections_received:2129
    total_commands_processed:317938954
    instantaneous_ops_per_sec:27
    rejected_connections:0
    sync_full:1
    sync_partial_ok:0
    sync_partial_err:1
    expired_keys:0
    evicted_keys:0
    keyspace_hits:38907578
    keyspace_misses:1114
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:7994

    # Replication
    role:master
    connected_slaves:1
    slave0:ip=10.18.0.51,port=6381,state=online,offset=138112056915,lag=0
    master_repl_offset:138116479593
    repl_backlog_active:1
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:138115431018
    repl_backlog_histlen:1048576

    # CPU
    used_cpu_sys:6054.23
    used_cpu_user:9857.41
    used_cpu_sys_children:0.09
    used_cpu_user_children:0.92

    # Commandstats
    cmdstat_set:calls=81145312,usec=205196176,usec_per_call=2.53
    cmdstat_del:calls=34985186,usec=715545862,usec_per_call=20.45
    cmdstat_mget:calls=1140733,usec=5482726,usec_per_call=4.81
    cmdstat_sadd:calls=122064614,usec=1332613141,usec_per_call=10.92
    cmdstat_srem:calls=1140735,usec=7931329,usec_per_call=6.95
    cmdstat_scard:calls=35449265,usec=122653140,usec_per_call=3.46
    cmdstat_spop:calls=18028,usec=41170,usec_per_call=2.28
    cmdstat_sdiffstore:calls=18194,usec=21805360,usec_per_call=1198.49
    cmdstat_smembers:calls=1177961,usec=83310181,usec_per_call=70.72
    cmdstat_ping:calls=1,usec=2,usec_per_call=2.00
    cmdstat_psync:calls=1,usec=8378,usec_per_call=8378.00
    cmdstat_replconf:calls=176387,usec=626705,usec_per_call=3.55
    cmdstat_flushdb:calls=2,usec=8609619,usec_per_call=4304809.50
    cmdstat_info:calls=12,usec=3347,usec_per_call=278.92
    cmdstat_eval:calls=40622523,usec=5067768003,usec_per_call=124.75

    # Keyspace
    db0:keys=44446,expires=0,avg_ttl=0
    hash_init_value: 1390085906

    [15035] 23 Jan 21:22:42.850 # --- CLIENT LIST OUTPUT
    [15035] 23 Jan 21:22:42.850 # addr=10.18.0.51:58992 fd=24 name= age=177162 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=replconf
    addr=10.18.0.31:34036 fd=8 name= age=11202 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34099 fd=10 name= age=11199 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34134 fd=15 name= age=11198 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34139 fd=16 name= age=11198 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34140 fd=17 name= age=11198 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34182 fd=18 name= age=11198 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34183 fd=19 name= age=11198 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34186 fd=20 name= age=11198 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34221 fd=22 name= age=11197 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47516 fd=31 name= age=11196 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47517 fd=32 name= age=11196 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47518 fd=33 name= age=11196 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34326 fd=35 name= age=11195 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47741 fd=44 name= age=11193 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47748 fd=45 name= age=11193 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47749 fd=46 name= age=11193 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47800 fd=47 name= age=11192 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47802 fd=48 name= age=11192 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47805 fd=49 name= age=11192 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:37837 fd=52 name= age=11164 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40421 fd=53 name= age=11137 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:48057 fd=37 name= age=2729 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=scard
    addr=10.18.0.33:48058 fd=39 name= age=2729 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:48682 fd=41 name= age=2707 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=1869295 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59704 fd=9 name= age=2127 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59707 fd=12 name= age=2127 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59980 fd=51 name= age=2125 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39441 fd=42 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39442 fd=43 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39443 fd=50 name= age=1736 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40200 fd=6 name= age=1427 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40203 fd=7 name= age=1427 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40217 fd=54 name= age=1427 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42166 fd=11 name= age=959 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42168 fd=13 name= age=959 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42536 fd=14 name= age=951 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33628 fd=26 name= age=933 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33629 fd=28 name= age=933 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33630 fd=34 name= age=933 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53237 fd=36 name= age=658 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53239 fd=38 name= age=658 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53247 fd=40 name= age=658 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:60248 fd=21 name= age=279 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=393783 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:60250 fd=23 name= age=279 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:33154 fd=25 name= age=267 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39626 fd=27 name= age=90 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39685 fd=29 name= age=90 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=5389218 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39895 fd=30 name= age=89 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd

    [15035] 23 Jan 21:22:42.851 # --- CURRENT CLIENT INFO
    [15035] 23 Jan 21:22:42.851 # client: addr=10.18.0.33:39442 fd=43 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    [15035] 23 Jan 21:22:42.851 # argv[0]: 'sadd'
    [15035] 23 Jan 21:22:42.851 # argv[1]: 'repo:5078656356b1021a550015bf:build:572750'
    [15035] 23 Jan 21:22:42.895 # argv[2]: <REDACTED>
    [15035] 23 Jan 21:22:42.895 # key 'repo:5078656356b1021a550015bf:build:572750' found in DB containing the following object:
    [15035] 23 Jan 21:22:42.895 # Object type: 2
    [15035] 23 Jan 21:22:42.895 # Object encoding: 2
    [15035] 23 Jan 21:22:42.895 # Object refcount: 1
    [15035] 23 Jan 21:22:42.895 # Set size: 19
    [15035] 23 Jan 21:22:42.895 # --- REGISTERS
    [15035] 23 Jan 21:22:42.895 #
    RAX:0000000000000000 RBX:000000000049e600
    RCX:00007f3b0212aeb0 RDX:0000000000000000
    RDI:00007f3b0212ae80 RSI:00000000ffffffff
    RBP:00000000004a7b3e RSP:00007fff1836e7e0
    R8 :0000000000000000 R9 :0000000000000000
    R10:0000000000004000 R11:0000000000000000
    R12:0000000000000b62 R13:0000000000000003
    R14:0000000000000001 R15:000000000000002b
    RIP:000000000044532f EFL:0000000000010202
    CSGSFS:0000000000000033
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ef) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ee) -> 00007fff1836e8b0
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ed) -> 00000000004261cc
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ec) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7eb) -> 00007f396c7e5000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ea) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e9) -> 000000000041e7a6
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e8) -> 00007f39d205de50
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e7) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e6) -> 00000000005d1bec
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e5) -> 000000000041fda9
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e4) -> 0000000000000003
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e3) -> 000000000041cfbc
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e2) -> 00000000005d1bf5
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e1) -> 0000000000000010
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e0) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # --- FAST MEMORY TEST
    [15035] 23 Jan 21:22:42.895 # Bio thread for job type #0 terminated
    [15035] 23 Jan 21:22:42.895 # Bio thread for job type #1 terminated
    Testing 6cc000 159744
    Testing b83000 1548288
    Testing 7f3940fff000 7495225344
    Testing 7f3affffd000 4096
    Testing 7f3afffff000 10485760
    Testing 7f3b00a00000 18874368
    Testing 7f3b0212a000 20480
    Testing 7f3b02348000 16384
    Testing 7f3b029eb000 16384
    Testing 7f3b029f5000 8192
    [15035] 23 Jan 21:24:31.678 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
    [15035] 23 Jan 21:24:31.678 #
    === REDIS BUG REPORT END. Make sure to include from START to END. ===
    [15035] 23 Jan 21:22:42.807 # Out Of Memory allocating 6102005 bytes!
    [15035] 23 Jan 21:22:42.817 #

    === REDIS BUG REPORT START: Cut & paste starting from here ===
    [15035] 23 Jan 21:22:42.817 # ------------------------------------------------
    [15035] 23 Jan 21:22:42.818 # !!! Software Failure. Press left mouse button to continue
    [15035] 23 Jan 21:22:42.818 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #redis.c:2914
    [15035] 23 Jan 21:22:42.818 # (forcing SIGSEGV in order to print the stack trace)
    [15035] 23 Jan 21:22:42.818 # ------------------------------------------------
    [15035] 23 Jan 21:22:42.818 #     Redis 2.8.2 crashed by signal: 11
    [15035] 23 Jan 21:22:42.818 #     Failed assertion: <no assertion failed> (<no file>:0)
    [15035] 23 Jan 21:22:42.818 # --- STACK TRACE
    /opt/redis/bin/redis-server 0.0.0.0:6381(logStackTrace+0x43)[0x445813]
    /opt/redis/bin/redis-server 0.0.0.0:6381(_redisPanic+0x7f)[0x44532f]
    /lib64/libpthread.so.0(+0xf710)[0x7f3b0213e710]
    /opt/redis/bin/redis-server 0.0.0.0:6381(_redisPanic+0x7f)[0x44532f]
    /opt/redis/bin/redis-server 0.0.0.0:6381(redisOutOfMemoryHandler+0x2c)[0x41cfbc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(zrealloc+0xf9)[0x41fda9]
    /opt/redis/bin/redis-server 0.0.0.0:6381(sdsMakeRoomFor+0x56)[0x41e7a6]
    /opt/redis/bin/redis-server 0.0.0.0:6381(processMultibulkBuffer+0x30c)[0x4261cc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(processInputBuffer+0x82)[0x426492]
    /opt/redis/bin/redis-server 0.0.0.0:6381(readQueryFromClient+0xb8)[0x426588]
    /opt/redis/bin/redis-server 0.0.0.0:6381(aeProcessEvents+0x13c)[0x4164dc]
    /opt/redis/bin/redis-server 0.0.0.0:6381(aeMain+0x2b)[0x41679b]
    /opt/redis/bin/redis-server 0.0.0.0:6381(main+0x2bf)[0x41ddbf]
    /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f3b01db9d1d]
    /opt/redis/bin/redis-server 0.0.0.0:6381[0x415af9]
    [15035] 23 Jan 21:22:42.849 # --- INFO OUTPUT
    [15035] 23 Jan 21:22:42.850 # # Server
    redis_version:2.8.2
    redis_git_sha1:00000000
    redis_git_dirty:0
    redis_build_id:c820c8bf89592140
    redis_mode:standalone
    os:Linux 2.6.32-042stab079.5 x86_64
    arch_bits:64
    multiplexing_api:epoll
    gcc_version:4.4.7
    process_id:15035
    run_id:b34ff247f5ff4ba0a792e146fe9efdd2cf271b56
    tcp_port:6381
    uptime_in_seconds:177185
    uptime_in_days:2
    hz:10
    lru_clock:639184
    config_file:/etc/redis/6381.conf

    # Clients
    connected_clients:48
    client_longest_output_list:0
    client_biggest_input_buf:1869295
    blocked_clients:0

    # Memory
    used_memory:6228251792
    used_memory_human:5.80G
    used_memory_rss:4386205696
    used_memory_peak:6461639456
    used_memory_peak_human:6.02G
    used_memory_lua:48128
    mem_fragmentation_ratio:0.70
    mem_allocator:jemalloc-3.2.0

    # Persistence
    loading:0
    rdb_changes_since_last_save:1051239090
    rdb_bgsave_in_progress:0
    rdb_last_save_time:1390335001
    rdb_last_bgsave_status:ok
    rdb_last_bgsave_time_sec:1
    rdb_current_bgsave_time_sec:-1
    aof_enabled:0
    aof_rewrite_in_progress:0
    aof_rewrite_scheduled:0
    aof_last_rewrite_time_sec:-1
    aof_current_rewrite_time_sec:-1
    aof_last_bgrewrite_status:ok

    # Stats
    total_connections_received:2129
    total_commands_processed:317938954
    instantaneous_ops_per_sec:27
    rejected_connections:0
    sync_full:1
    sync_partial_ok:0
    sync_partial_err:1
    expired_keys:0
    evicted_keys:0
    keyspace_hits:38907578
    keyspace_misses:1114
    pubsub_channels:0
    pubsub_patterns:0
    latest_fork_usec:7994

    # Replication
    role:master
    connected_slaves:1
    slave0:ip=10.18.0.51,port=6381,state=online,offset=138112056915,lag=0
    master_repl_offset:138116479593
    repl_backlog_active:1
    repl_backlog_size:1048576
    repl_backlog_first_byte_offset:138115431018
    repl_backlog_histlen:1048576

    # CPU
    used_cpu_sys:6054.23
    used_cpu_user:9857.41
    used_cpu_sys_children:0.09
    used_cpu_user_children:0.92

    # Commandstats
    cmdstat_set:calls=81145312,usec=205196176,usec_per_call=2.53
    cmdstat_del:calls=34985186,usec=715545862,usec_per_call=20.45
    cmdstat_mget:calls=1140733,usec=5482726,usec_per_call=4.81
    cmdstat_sadd:calls=122064614,usec=1332613141,usec_per_call=10.92
    cmdstat_srem:calls=1140735,usec=7931329,usec_per_call=6.95
    cmdstat_scard:calls=35449265,usec=122653140,usec_per_call=3.46
    cmdstat_spop:calls=18028,usec=41170,usec_per_call=2.28
    cmdstat_sdiffstore:calls=18194,usec=21805360,usec_per_call=1198.49
    cmdstat_smembers:calls=1177961,usec=83310181,usec_per_call=70.72
    cmdstat_ping:calls=1,usec=2,usec_per_call=2.00
    cmdstat_psync:calls=1,usec=8378,usec_per_call=8378.00
    cmdstat_replconf:calls=176387,usec=626705,usec_per_call=3.55
    cmdstat_flushdb:calls=2,usec=8609619,usec_per_call=4304809.50
    cmdstat_info:calls=12,usec=3347,usec_per_call=278.92
    cmdstat_eval:calls=40622523,usec=5067768003,usec_per_call=124.75

    # Keyspace
    db0:keys=44446,expires=0,avg_ttl=0
    hash_init_value: 1390085906

    [15035] 23 Jan 21:22:42.850 # --- CLIENT LIST OUTPUT
    [15035] 23 Jan 21:22:42.850 # addr=10.18.0.51:58992 fd=24 name= age=177162 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=replconf
    addr=10.18.0.31:34036 fd=8 name= age=11202 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34099 fd=10 name= age=11199 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34134 fd=15 name= age=11198 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34139 fd=16 name= age=11198 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34140 fd=17 name= age=11198 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34182 fd=18 name= age=11198 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34183 fd=19 name= age=11198 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34186 fd=20 name= age=11198 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34221 fd=22 name= age=11197 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47516 fd=31 name= age=11196 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47517 fd=32 name= age=11196 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47518 fd=33 name= age=11196 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:34326 fd=35 name= age=11195 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47741 fd=44 name= age=11193 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47748 fd=45 name= age=11193 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47749 fd=46 name= age=11193 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47800 fd=47 name= age=11192 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47802 fd=48 name= age=11192 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:47805 fd=49 name= age=11192 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:37837 fd=52 name= age=11164 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40421 fd=53 name= age=11137 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:48057 fd=37 name= age=2729 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=scard
    addr=10.18.0.33:48058 fd=39 name= age=2729 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:48682 fd=41 name= age=2707 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=1869295 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59704 fd=9 name= age=2127 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59707 fd=12 name= age=2127 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:59980 fd=51 name= age=2125 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39441 fd=42 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39442 fd=43 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39443 fd=50 name= age=1736 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40200 fd=6 name= age=1427 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40203 fd=7 name= age=1427 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:40217 fd=54 name= age=1427 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42166 fd=11 name= age=959 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42168 fd=13 name= age=959 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:42536 fd=14 name= age=951 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33628 fd=26 name= age=933 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33629 fd=28 name= age=933 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:33630 fd=34 name= age=933 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53237 fd=36 name= age=658 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53239 fd=38 name= age=658 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:53247 fd=40 name= age=658 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:60248 fd=21 name= age=279 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=393783 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:60250 fd=23 name= age=279 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.31:33154 fd=25 name= age=267 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39626 fd=27 name= age=90 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39685 fd=29 name= age=90 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=5389218 obl=0 oll=0 omem=0 events=r cmd=sadd
    addr=10.18.0.33:39895 fd=30 name= age=89 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd

    [15035] 23 Jan 21:22:42.851 # --- CURRENT CLIENT INFO
    [15035] 23 Jan 21:22:42.851 # client: addr=10.18.0.33:39442 fd=43 name= age=1736 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
    [15035] 23 Jan 21:22:42.851 # argv[0]: 'sadd'
    [15035] 23 Jan 21:22:42.851 # argv[1]: 'repo:5078656356b1021a550015bf:build:572750'
    [15035] 23 Jan 21:22:42.895 # argv[2]: <REDACTED>
    [15035] 23 Jan 21:22:42.895 # key 'repo:5078656356b1021a550015bf:build:572750' found in DB containing the following object:
    [15035] 23 Jan 21:22:42.895 # Object type: 2
    [15035] 23 Jan 21:22:42.895 # Object encoding: 2
    [15035] 23 Jan 21:22:42.895 # Object refcount: 1
    [15035] 23 Jan 21:22:42.895 # Set size: 19
    [15035] 23 Jan 21:22:42.895 # --- REGISTERS
    [15035] 23 Jan 21:22:42.895 #
    RAX:0000000000000000 RBX:000000000049e600
    RCX:00007f3b0212aeb0 RDX:0000000000000000
    RDI:00007f3b0212ae80 RSI:00000000ffffffff
    RBP:00000000004a7b3e RSP:00007fff1836e7e0
    R8 :0000000000000000 R9 :0000000000000000
    R10:0000000000004000 R11:0000000000000000
    R12:0000000000000b62 R13:0000000000000003
    R14:0000000000000001 R15:000000000000002b
    RIP:000000000044532f EFL:0000000000010202
    CSGSFS:0000000000000033
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ef) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ee) -> 00007fff1836e8b0
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ed) -> 00000000004261cc
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ec) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7eb) -> 00007f396c7e5000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7ea) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e9) -> 000000000041e7a6
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e8) -> 00007f39d205de50
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e7) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e6) -> 00000000005d1bec
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e5) -> 000000000041fda9
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e4) -> 0000000000000003
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e3) -> 000000000041cfbc
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e2) -> 00000000005d1bf5
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e1) -> 0000000000000010
    [15035] 23 Jan 21:22:42.895 # (00007fff1836e7e0) -> 0000000000000000
    [15035] 23 Jan 21:22:42.895 # --- FAST MEMORY TEST
    [15035] 23 Jan 21:22:42.895 # Bio thread for job type #0 terminated
    [15035] 23 Jan 21:22:42.895 # Bio thread for job type #1 terminated
    Testing 6cc000 159744
    Testing b83000 1548288
    Testing 7f3940fff000 7495225344
    Testing 7f3affffd000 4096
    Testing 7f3afffff000 10485760
    Testing 7f3b00a00000 18874368
    Testing 7f3b0212a000 20480
    Testing 7f3b02348000 16384
    Testing 7f3b029eb000 16384
    Testing 7f3b029f5000 8192
    [15035] 23 Jan 21:24:31.678 # Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
    [15035] 23 Jan 21:24:31.678 #
    === REDIS BUG REPORT END. Make sure to include from START to END. ===
@badboy

Could you pleas provide the config? How much memory does your machine have?

@charsyam

@noahd1 How much memory does your machine have? I might think it is not because of memory policy.

@noahd1

The VM (it's a open vz VM) has 8GB of total memory. The redis config is below:

    daemonize no
    pidfile /var/run/redis/6381.pid
    port 6381
    bind 0.0.0.0
    timeout 300
    tcp-keepalive 0
    loglevel notice
    logfile ""
    syslog-enabled yes
    syslog-ident redis-6381
    databases 16
    save ""
    stop-writes-on-bgsave-error yes
    rdbcompression yes
    rdbchecksum yes
    dbfilename dump.rdb
    dir /srv/data/redis/6381
    slave-serve-stale-data yes
    slave-read-only yes
    repl-disable-tcp-nodelay no
    slave-priority 100
    maxmemory 6gb
    maxmemory-policy noeviction
    appendonly no
    appendfilename appendonly.aof
    appendfsync everysec
    no-appendfsync-on-rewrite no
    auto-aof-rewrite-percentage 100
    auto-aof-rewrite-min-size 64mb
    lua-time-limit 5000
    slowlog-log-slower-than 10000
    slowlog-max-len 1024
    notify-keyspace-events ""
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-entries 512
    list-max-ziplist-value 64
    set-max-intset-entries 512
    zset-max-ziplist-entries 128
    zset-max-ziplist-value 64
    activerehashing yes
    client-output-buffer-limit normal 0 0 0
    client-output-buffer-limit slave 256mb 64mb 60
    client-output-buffer-limit pubsub 32mb 8mb 60
    hz 10
    aof-rewrite-incremental-fsync yes
@charsyam

@noahd1 could you show me result of "sysctl vm.overcommit_memory"
maybe it is 0,
and Did you run redis only your server? I think it is because of lacking memory.

@noahd1

Hi @charsyam and others. Thanks for your responses. It could very well be because of memory on the VM and not the max memory policy. This wasn't my initial suspicion because the memory it was using was close to the configured redis max memory limit.

To answer your question .. the overcommit_memory setting is set to "0" unfortunately. We're not using bare metal and was told that our virtualization of choice doesn't support this setting even if it were set to "1".

I'm sure these two together paint the picture you need, but I'm certainly not an expert on linux memory usage to say what specifically. Would you mind crystallizing it for me?

@badboy

Your redis used about 6GB out of 8GB available, so if other programs used the rest 2GB this crash to be expected, even though you have set a maxmemory policy.

@charsyam

Hi. @noahd1
Thank you for your informing.
Maybe it is better to check your system.

There is some possibilities.
running other processes(backup process or monitoring, other things)
and your setting is vm.overcommit_memory and your swap memory will be 0.
so request can be failed

so. Just always check your server with memory usage.
and when it needs more memory, removing some items or scale up to other servers.

@noahd1

Thanks everyone. Closing this out. We found/reduced the memory usage in Redis so don't expect this to come up anytime soon again. We have 3 total instances of redis on this VM, so it likely just ran out of memory.

@noahd1 noahd1 closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.