[Question] CouchDB crashed during benchmark

[style95]

CouchDB crashed during benchmarking.

I deployed 3 nodes.

• node1: 10.113.130.91
• node2: 10.113.130.92
• node3: 10.113.130.93

I sent 500 docs using bulk-insert API.

It showed steady performance, and after about 4 minutes, suddenly one of nodes crashed.

I got following logs on the nodes.

node1

[error] 2017-12-18T03:13:09.183711Z couchdb@10.113.130.91 emulator -------- Error in process <0.21270.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[error] 2017-12-18T03:13:09.183768Z couchdb@10.113.130.91 emulator -------- Error in process <0.21304.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[warning] 2017-12-18T03:13:09.183766Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/00000000-1fffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[error] 2017-12-18T03:13:09.183804Z couchdb@10.113.130.91 emulator -------- Error in process <0.21269.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[warning] 2017-12-18T03:13:09.183981Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/00000000-1fffffff/_global_changes.1513565315 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[warning] 2017-12-18T03:13:09.184206Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/60000000-7fffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[error] 2017-12-18T03:13:09.186470Z couchdb@10.113.130.91 emulator -------- Error in process <0.21343.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[error] 2017-12-18T03:13:09.186529Z couchdb@10.113.130.91 emulator -------- Error in process <0.21372.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[warning] 2017-12-18T03:13:09.186662Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/80000000-9fffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[warning] 2017-12-18T03:13:09.186962Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/c0000000-dfffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[error] 2017-12-18T03:13:09.188375Z couchdb@10.113.130.91 emulator -------- Error in process <0.21432.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[error] 2017-12-18T03:13:09.188415Z couchdb@10.113.130.91 emulator -------- Error in process <0.21424.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[warning] 2017-12-18T03:13:09.188476Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/e0000000-ffffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[warning] 2017-12-18T03:13:09.188649Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/20000000-3fffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[error] 2017-12-18T03:13:09.190738Z couchdb@10.113.130.91 emulator -------- Error in process <0.21483.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[error] 2017-12-18T03:13:09.190787Z couchdb@10.113.130.91 emulator -------- Error in process <0.21489.8> on node 'couchdb@10.113.130.91' with exit value: {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,"src/mem3_rpc.erl"},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,"src/mem3_rep.erl"},{line,194}]},{mem3_rep,repl,2,[{file,"src/mem3_rep.erl"},...
[warning] 2017-12-18T03:13:09.190856Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/40000000-5fffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}
[warning] 2017-12-18T03:13:09.191076Z couchdb@10.113.130.91 <0.294.0> -------- mem3_sync shards/a0000000-bfffffff/lambda-bmt_activations.1513565327 couchdb@10.113.130.92 {{rexi_DOWN,{'couchdb@10.113.130.92',noconnect}},[{mem3_rpc,rexi_call,2,[{file,[115,114,99,47,109,101,109,51,95,114,112,99,46,101,114,108]},{line,269}]},{mem3_rep,calculate_start_seq,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,194}]},{mem3_rep,repl,2,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,175}]},{mem3_rep,go,1,[{file,[115,114,99,47,109,101,109,51,95,114,101,112,46,101,114,108]},{line,81}]},{mem3_sync,'-start_push_replication/1-fun-0-',2,[{file,[115,114,99,47,109,101,109,51,95,115,121,110,99,46,101,114,108]},{line,208}]}]}

node2 crashed.
But I am not sure what the problem is.

node2

[error] 2017-12-18T03:12:54.538612Z couchdb@10.113.130.92 <0.1329.0> -------- OS Process died with status: 137
[error] 2017-12-18T03:12:54.549992Z couchdb@10.113.130.92 <0.1329.0> -------- gen_server <0.1329.0> terminated with reason: {exit_status,137}
  last msg: {#Port<0.8125>,{exit_status,137}}
     state: {os_proc,"./bin/couchjs ./share/server/main.js",#Port<0.8125>,#Fun<couch_os_process.writejson.2>,#Fun<couch_os_process.readjson.1>,5000,300000}
[info] 2017-12-18T03:12:54.551745Z couchdb@10.113.130.92 <0.213.0> -------- couch_proc_manager <0.1329.0> died {exit_status,137}
[error] 2017-12-18T03:12:54.552261Z couchdb@10.113.130.92 <0.1329.0> -------- CRASH REPORT Process  (<0.1329.0>) with 0 neighbors exited with reason: {exit_status,137} at gen_server:terminate/6(line:737) <= proc_lib:init_p_do_apply/3(line:237); initial_call: {couch_os_process,init,['Argument__1']}, ancestors: [<0.1328.0>], messages: [], links: [<0.213.0>], dictionary: [], trap_exit: false, status: running, heap_size: 610, stack_size: 27, reductions: 2890

And I could occasionally observed following error as well.
I wonder why one of my nodes is suddenly disallowed.

[error] 2017-12-18T03:13:06.373063Z couchdb@10.113.130.91 <0.16717.8> -------- ** Connection attempt from disallowed node 'couchdb@10.113.130.92' **

Though there are high loads, I expect couchdb reject requests rather than crashed.
But I don't know how to configure it.

I used default configuration with following changes.

default.ini

max_dbs_open = 1024
os_process_limit = 2000
os_process_soft_limit = 1500
check_interval = 60
_default = [{db_fragmentation, "30%"}, {view_fragmentation, "30%"}]

vm.args

+A 1024

I already looked into couchDB guide and I could know what the configurations are, but it was not easy for me to figure out what is the proper value for each configurations.

Could anyone guide me to tune it for production? or are there any guide to tune CouchDB?
It would be great to have some recommendation such as os_process_limit = CPU cores * 10, +A = CPU cores * 15 and so on.

I am using following machines:

• CPU: 32 cores
• MEM: 64 GB
• DISK: 1.3 TB SSD
• Network: 1G

[style95]

I found, memory usage of one node1 keep increasing until 95%, and suddenly all modules are restarted.

Memory usage of other nodes(node2, 3) were stable under 40%.

[info] 2017-12-18T06:04:11.965787Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_log started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:11.969249Z couchdb@10.113.130.91 <0.7.0> -------- Application folsom started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:11.998677Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_stats started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:11.998838Z couchdb@10.113.130.91 <0.7.0> -------- Application khash started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.005142Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_event started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.009451Z couchdb@10.113.130.91 <0.7.0> -------- Application ibrowse started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.013053Z couchdb@10.113.130.91 <0.7.0> -------- Application ioq started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.013204Z couchdb@10.113.130.91 <0.7.0> -------- Application mochiweb started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.018868Z couchdb@10.113.130.91 <0.201.0> -------- Apache CouchDB 2.1.1 is starting.

[info] 2017-12-18T06:04:12.018999Z couchdb@10.113.130.91 <0.202.0> -------- Starting couch_sup
[info] 2017-12-18T06:04:12.140996Z couchdb@10.113.130.91 <0.201.0> -------- Apache CouchDB has started. Time to relax.

[info] 2017-12-18T06:04:12.141142Z couchdb@10.113.130.91 <0.201.0> -------- Apache CouchDB has started on http://any:5986/
[info] 2017-12-18T06:04:12.141225Z couchdb@10.113.130.91 <0.7.0> -------- Application couch started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.141311Z couchdb@10.113.130.91 <0.7.0> -------- Application ets_lru started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.163563Z couchdb@10.113.130.91 <0.7.0> -------- Application rexi started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.165314Z couchdb@10.113.130.91 <0.265.0> -------- Opening index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
[info] 2017-12-18T06:04:12.177415Z couchdb@10.113.130.91 <0.291.0> -------- Opening index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
[error] 2017-12-18T06:04:12.197986Z couchdb@10.113.130.91 emulator -------- Error in process <0.308.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<19 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.198169Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `subjects` from database `shards/e0000000-ffffffff/lambda-bmt_subjects.1513576321`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.209860Z couchdb@10.113.130.91 emulator -------- Error in process <0.343.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<22 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.210001Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `logCleanup` from database `shards/c0000000-dfffffff/lambda-bmt_activations.1513576331`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_activations">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.212308Z couchdb@10.113.130.91 emulator -------- Error in process <0.350.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<11 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.212430Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `_replicator` from database `shards/80000000-9fffffff/_replicator.1513576319`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_replicator">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.218714Z couchdb@10.113.130.91 emulator -------- Error in process <0.366.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<22 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.218968Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `whisks-filters.v2` from database `shards/80000000-9fffffff/lambda-bmt_activations.1513576331`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_activations">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.226318Z couchdb@10.113.130.91 emulator -------- Error in process <0.391.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<6 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.226595Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `_auth` from database `shards/40000000-5fffffff/_users.1513576319`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.236714Z couchdb@10.113.130.91 emulator -------- Error in process <0.425.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<17 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.236955Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `whisks.v2` from database `shards/20000000-3fffffff/lambda-bmt_whisks.1513576330`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_whisks">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.239656Z couchdb@10.113.130.91 emulator -------- Error in process <0.429.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<22 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.239796Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `whisks.v2` from database `shards/20000000-3fffffff/lambda-bmt_activations.1513576331`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_activations">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.243468Z couchdb@10.113.130.91 emulator -------- Error in process <0.442.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<19 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.243604Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `snapshotFilters` from database `shards/00000000-1fffffff/lambda-bmt_subjects.1513576321`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_subjects">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.245585Z couchdb@10.113.130.91 emulator -------- Error in process <0.449.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<17 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.245828Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `snapshotFilters` from database `shards/00000000-1fffffff/lambda-bmt_whisks.1513576330`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_whisks">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.248764Z couchdb@10.113.130.91 emulator -------- Error in process <0.453.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<22 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.248891Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `snapshotFilters` from database `shards/00000000-1fffffff/lambda-bmt_activations.1513576331`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_activations">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[info] 2017-12-18T06:04:12.250156Z couchdb@10.113.130.91 <0.456.0> -------- Starting compaction for db "shards/00000000-1fffffff/_global_changes.1513576319"
[notice] 2017-12-18T06:04:12.259997Z couchdb@10.113.130.91 <0.456.0> -------- Compaction swap for db: ./data/shards/00000000-1fffffff/_global_changes.1513576319.couch 5509326 49352
[info] 2017-12-18T06:04:12.260331Z couchdb@10.113.130.91 <0.456.0> -------- Compaction for db "shards/00000000-1fffffff/_global_changes.1513576319" completed.
[error] 2017-12-18T06:04:12.260952Z couchdb@10.113.130.91 emulator -------- Error in process <0.465.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<15 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.265586Z couchdb@10.113.130.91 emulator -------- Error in process <0.481.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<16 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.265820Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `namespaces` from database `shards/a0000000-bfffffff/service_mappings.1513576326`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"service_mappings">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2017-12-18T06:04:12.269434Z couchdb@10.113.130.91 emulator -------- Error in process <0.488.0> on node 'couchdb@10.113.130.91' with exit value: {badarg,[{ets,member,[mem3_openers,<<22 bytes>>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk...
[error] 2017-12-18T06:04:12.269561Z couchdb@10.113.130.91 <0.254.0> -------- Error opening view group `activations` from database `shards/a0000000-bfffffff/lambda-bmt_activations.1513576331`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"lambda-bmt_activations">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,184}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,194}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[notice] 2017-12-18T06:04:12.379628Z couchdb@10.113.130.91 <0.269.0> -------- rexi_server_mon : cluster unstable
[notice] 2017-12-18T06:04:12.379672Z couchdb@10.113.130.91 <0.272.0> -------- rexi_server_mon : cluster unstable
[notice] 2017-12-18T06:04:12.379798Z couchdb@10.113.130.91 <0.267.0> -------- rexi_server : cluster unstable
[notice] 2017-12-18T06:04:12.379926Z couchdb@10.113.130.91 <0.271.0> -------- rexi_buffer : cluster unstable
[notice] 2017-12-18T06:04:12.404666Z couchdb@10.113.130.91 <0.269.0> -------- rexi_server_mon : cluster unstable
[notice] 2017-12-18T06:04:12.404757Z couchdb@10.113.130.91 <0.272.0> -------- rexi_server_mon : cluster unstable
[notice] 2017-12-18T06:04:12.404825Z couchdb@10.113.130.91 <0.267.0> -------- rexi_server : cluster unstable
[notice] 2017-12-18T06:04:12.404877Z couchdb@10.113.130.91 <0.271.0> -------- rexi_buffer : cluster unstable
[info] 2017-12-18T06:04:12.417694Z couchdb@10.113.130.91 <0.7.0> -------- Application mem3 started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.417779Z couchdb@10.113.130.91 <0.7.0> -------- Application fabric started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.478630Z couchdb@10.113.130.91 <0.7.0> -------- Application chttpd started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.529773Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_index started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.529875Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_mrview started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.529942Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_plugins started on node 'couchdb@10.113.130.91'
[notice] 2017-12-18T06:04:12.773666Z couchdb@10.113.130.91 <0.88.0> -------- config: [features] scheduler set to true for reason nil
[notice] 2017-12-18T06:04:12.839556Z couchdb@10.113.130.91 <0.874.0> 96b227aa8c 10.113.130.91:5984 10.66.100.4 admin GET /_active_tasks 200 ok 90
[info] 2017-12-18T06:04:12.967585Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_replicator started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:12.984160Z couchdb@10.113.130.91 <0.7.0> -------- Application couch_peruser started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.005258Z couchdb@10.113.130.91 <0.7.0> -------- Application ddoc_cache started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.031852Z couchdb@10.113.130.91 <0.7.0> -------- Application global_changes started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.032102Z couchdb@10.113.130.91 <0.7.0> -------- Application jiffy started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.045481Z couchdb@10.113.130.91 <0.7.0> -------- Application mango started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.055213Z couchdb@10.113.130.91 <0.7.0> -------- Application setup started on node 'couchdb@10.113.130.91'
[info] 2017-12-18T06:04:13.055272Z couchdb@10.113.130.91 <0.7.0> -------- Application snappy started on node 'couchdb@10.113.130.91

[style95]

I found memory usage keep increasing til 90~95% and at some point, kernel killed beam.smp, due to out of memory issue.

[wohali]

So in the first paste, CouchDB is saying that an external OS process (couchjs) terminated with exit code 137.

137=128+9, which means some other process has sent you a signal 9, which is SIGKILL. Presumably the kernel oomkiller killed your couchjs process.

My guess? It's your alteration of +A and os_process_limit.

Do not modify +A to such a large number. Our default of +A 16 is correct for most machines, only needing to be lowered for smaller machines. (The default Erlang value is 10.) Read more about it here: https://stackoverflow.com/questions/13479142/when-is-it-appropriate-to-increase-the-async-thread-size-from-zero#13497717

The os_process_limit value defines how many couchjs processes can run at once. Typically you want this to be # of cores - 1 at most (leaving a core for Erlang and I/O handling), but sometimes less, if you are running more than just CouchDB on a machine (say, the clouseau/dreyfus full-text indexing functionality.)

[style95]

The reason why I increased the value is because I am facing this issue: #1073
Is there any way to prevent this issue?

[wohali]

Let's continue the discussion over there...but the tweaks you applied aren't appropriate for the environment you're running in.