Riak 1.3 suddenly crashes, even without a big load

[subnetmarco]

I've got the following problem when running Riak 1.3 with LevelDB on EC2 (c1.xlarge):

$ sudo cat /var/log/riak/error.log 
2013-03-28 19:18:35.943 [error] <0.88.0> gen_server disksup terminated with reason: {port_died,normal}

$ sudo cat /var/log/riak/crash.log
2013-03-28 19:18:36 =ERROR REPORT====
** Generic server disksup terminating 
** Last message in was {'EXIT',#Port<0.1415>,normal}
** When Server state == [{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/",206424760,29},{"/dev/shm",3559560,0}]}]}]
** Reason for termination == 
** {port_died,normal}
2013-03-28 19:18:37 =CRASH REPORT====
  crasher:
    initial call: disksup:init/1
    pid: <0.88.0>
    registered_name: disksup
    exception exit: {{port_died,normal},[{gen_server,terminate,6,[{file,"gen_server.erl"},{line,747}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
    ancestors: [os_mon_sup,<0.86.0>]
    messages: []
    links: [<0.87.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 98287
  neighbours:

This happens suddenly also when the cluster is not stressed. The cluster is made of 4 machines, 64 partitions, and n_val=2. This is a very sever problem because it often happens that the whole cluster crashes. Each server has an average of 60GB of data stored.

These are other logs from another machine in the same cluster, that crashed as well:

• error.log: http://pastebin.com/ARNqLE3z
• crash.log: http://pastebin.com/WPUF2wjW

LevelDB is configured like this on every machine:

%% eLevelDB Config
{eleveldb, [
    {data_root, "/var/lib/riak/leveldb"},
    {max_open_files, 50}, %% Maximum number of files open at once per partition
    {cache_size, 117440512} %% 112MB
]}

The machines are EC2 High CPU extra large instances (c1.xlarge), as such they have:

• 7 GiB of memory
• 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)

I'm using the Erlang VM bundled with the RHEL 6.0 x86 64 distribution.

[subnetmarco]

The cluster crashed again, this time the I've been able to log the following error:

** Generic server disksup terminating 
** Last message in was timeout
** When Server state == {state,80,1800000,{unix,linux},[],#Port<0.448653>}
** Reason for termination == 
** {badarg,[{erlang,port_close,[#Port<0.448653>],[]},{disksup,terminate,2,[{file,"disksup.erl"},{line,164}]},{gen_server,terminate,6,[{file,"gen_server.erl"},{line,722}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
2013-04-02 12:12:39 =CRASH REPORT====
  crasher:
    initial call: disksup:init/1
    pid: <0.7375.222>
    registered_name: disksup
    exception exit: {{badarg,[{erlang,port_close,[#Port<0.448653>],[]},{disksup,terminate,2,[{file,"disksup.erl"},{line,164}]},{gen_server,terminate,6,[{file,"gen_server.erl"},{line,722}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},[{gen_server,terminate,6,[{file,"gen_server.erl"},{line,725}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
    ancestors: [os_mon_sup,<0.81.0>]
    messages: []
    links: [<0.82.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 610
    stack_size: 24
    reductions: 312
  neighbours:
2013-04-02 12:12:39 =SUPERVISOR REPORT====
     Supervisor: {local,os_mon_sup}
     Context:    child_terminated
     Reason:     {badarg,[{erlang,port_close,[#Port<0.448653>],[]},{disksup,terminate,2,[{file,"disksup.erl"},{line,164}]},{gen_server,terminate,6,[{file,"gen_server.erl"},{line,722}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
     Offender:   [{pid,<0.7375.222>},{name,disksup},{mfargs,{disksup,start_link,[]}},{restart_type,permanent},{shutdown,2000},{child_type,worker}]

[subnetmarco]

After updating to 1.3.1 the whole cluster crashed again. Now I've been able to log the following error, which suggests a lack of memory on the server. I wonder how can I then limit memory consumption.

2013-04-08 11:34:28 =SUPERVISOR REPORT====
     Supervisor: {local,riak_pipe_builder_sup}
     Context:    child_terminated
     Reason:     {error,insufficient_vnodes_available}
     Offender:   [{pid,<0.10710.548>},{name,undefined},{mfargs,{riak_pipe_builder,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}]

2013-04-08 13:20:54 =ERROR REPORT====
** Generic server disksup terminating 
** Last message in was {'EXIT',#Port<0.117611876>,normal}
** When Server state == [{data,[{"OS",{unix,linux}},{"Timeout",1800000},{"Threshold",80},{"DiskData",[{"/",206424760,29},{"/dev/shm",3559472,0}]}]}]
** Reason for termination == 
** {port_died,normal}
2013-04-08 13:20:54 =ERROR REPORT====
** Generic server memsup terminating 
** Last message in was {'EXIT',<0.85.0>,{port_died,normal}}
** When Server state == [{data,[{"Timeout",60000}]},{items,{"Memory Usage",[{"Allocated",7256293376},{"Total",7289798656}]}},{items,{"Worst Memory User",[{"Pid",<0.141.0>},{"Memory",4114672}]}}]
** Reason for termination == 
** {port_died,normal}
2013-04-08 13:20:54 =CRASH REPORT====
  crasher:
    initial call: memsup:init/1
    pid: <0.84.0>
    registered_name: memsup
    exception exit: {{port_died,normal},[{gen_server,terminate,6,[{file,"gen_server.erl"},{line,747}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
    ancestors: [os_mon_sup,<0.81.0>]
    messages: []
    links: [<0.82.0>]
    dictionary: [{system_memory_high_watermark,set}]
    trap_exit: true
    status: running
    heap_size: 377
    stack_size: 24
    reductions: 2613311929
  neighbours:

[jaredmorrow]

Since replies have been made on the mailing-list, I'm going to go ahead and close this as an issue for now.

[peopleforrester]

Jaredmorrow,
I am having the exact same issue and I am on the mailing list as well. Can you refer me to the thread that gave you some insight as to the answer.

thx.

[subnetmarco]

@mmajere http://riak-users.197444.n3.nabble.com/Unexpected-Riak-1-3-crash-td4027359.html