all_dbs_active error when max_dbs_open set very low crashes rexi, returns 500

[wohali]

I'm fixing up the stats.js test, which intentionally sets max_dbs_open low (but unintentionally lower than q). When an operation (here, a doc creation) hits all_dbs_active, rexi crashes with a badmatch, which results in a 500 being returned for the PUT operation.

Log:

    [notice] 2017-05-02T21:48:23.896392Z node1@127.0.0.1 <0.310.0> e0d886f91b 127.0.0.1:15984 127.0.0.1 undefined GET / 200 ok 0
    [notice] 2017-05-02T21:48:23.896987Z node1@127.0.0.1 <0.310.0> 71587dc11d 127.0.0.1:15984 127.0.0.1 undefined GET /_membership 200 ok 0
    [notice] 2017-05-02T21:48:23.934277Z node1@127.0.0.1 <0.69.0> -------- config: [couchdb] max_dbs_open set to 5 for reason nil
    [notice] 2017-05-02T21:48:23.934653Z node1@127.0.0.1 <0.310.0> b0c18fcc87 127.0.0.1:15984 127.0.0.1 undefined PUT /_node/node1@127.0.0.1/_config/couchdb/max_dbs_open 200 ok 37
    [notice] 2017-05-02T21:48:23.950708Z node1@127.0.0.1 <0.310.0> d7719381c7 127.0.0.1:15984 127.0.0.1 undefined GET /_node/node1@127.0.0.1/_stats/couchdb/open_databases?flush=true 200 ok 16
    [notice] 2017-05-02T21:48:23.967092Z node1@127.0.0.1 <0.310.0> 8257556df3 127.0.0.1:15984 127.0.0.1 undefined GET /_node/node1@127.0.0.1/_stats/couchdb/open_os_files?flush=true 200 ok 16
    [error] 2017-05-02T21:48:23.968115Z node1@127.0.0.1 <0.310.0> 8541329306 Request to create N=3 DB but only 1 node(s)
    [notice] 2017-05-02T21:48:24.147960Z node1@127.0.0.1 <0.310.0> 8541329306 127.0.0.1:15984 127.0.0.1 undefined PUT /test_suite_db_ougfuqun/ 201 ok 180
    [error] 2017-05-02T21:48:24.215579Z node1@127.0.0.1 <0.485.0> -------- rexi_server error:{badmatch,{error,all_dbs_active}} [{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
    [error] 2017-05-02T21:48:24.215937Z node1@127.0.0.1 <0.486.0> -------- rexi_server error:{badmatch,{error,all_dbs_active}} [{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
    [error] 2017-05-02T21:48:24.216063Z node1@127.0.0.1 <0.487.0> -------- rexi_server error:{badmatch,{error,all_dbs_active}} [{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]
    [error] 2017-05-02T21:48:24.216799Z node1@127.0.0.1 emulator -------- Error in process <0.479.0> on node 'node1@127.0.0.1' with exit value: {{badmatch,{error,{badmatch,{error,all_dbs_active},[{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]}}},[{ddoc_cache_opener,recover_validation_funs...
    [error] 2017-05-02T21:48:24.216898Z node1@127.0.0.1 emulator -------- Error in process <0.478.0> on node 'node1@127.0.0.1' with exit value: {{case_clause,{error,{{badmatch,{error,{badmatch,{error,all_dbs_active},[{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]}}},[{ddoc_cache_opener...
    [error] 2017-05-02T21:48:24.217563Z node1@127.0.0.1 <0.476.0> -------- could not load validation funs {{case_clause,{error,{{badmatch,{error,{badmatch,{error,all_dbs_active},[{fabric_rpc,all_docs,3,[{file,"src/fabric_rpc.erl"},{line,100}]},{rexi_server,init_p,3,[{file,"src/rexi_server.erl"},{line,139}]}]}}},[{ddoc_cache_opener,recover_validation_funs,1,[{file,"src/ddoc_cache_opener.erl"},{line,127}]},{ddoc_cache_opener,fetch_doc_data,1,[{file,"src/ddoc_cache_opener.erl"},{line,240}]}]}}},[{ddoc_cache_opener,handle_open_response,1,[{file,"src/ddoc_cache_opener.erl"},{line,282}]},{couch_db,'-load_validation_funs/1-fun-0-',1,[{file,"src/couch_db.erl"},{line,659}]}]}
    [notice] 2017-05-02T21:48:24.219008Z node1@127.0.0.1 <0.310.0> a4b1d64a3e 127.0.0.1:15984 127.0.0.1 undefined PUT /test_suite_db_ougfuqun/0 500 ok 68
    [error] 2017-05-02T21:48:24.224920Z node1@127.0.0.1 <0.489.0> -------- Could not open file /home/joant/couchdb/dev/lib/node1/data/shards/40000000-5fffffff/test_suite_db_ougfuqun.1493761703.couch: no such file or directory
    [info] 2017-05-02T21:48:24.225349Z node1@127.0.0.1 <0.217.0> -------- open_result error {not_found,no_db_file} for shards/40000000-5fffffff/test_suite_db_ougfuqun.1493761703
    [warning] 2017-05-02T21:48:24.225476Z node1@127.0.0.1 <0.483.0> -------- creating missing database: shards/40000000-5fffffff/test_suite_db_ougfuqun.1493761703

[wohali]

The error seems to occur especially when max_dbs_open is set extremely low, in this case < q. This is an edge case that we probably don't need to spend too much time on.

Removing this from the 2.1.0 milestone.

[wohali]

I don't think keeping this ticket open is valuable. If you're dumb enough to set max_dbs_open smaller than q for a single database, you deserve a crash.

[nintra]

I guess I never changed this value in the configuration and after some months in production the count of user dbs was higher than this value. My logs were flooded with this "badmatch" errors. And I had no clue how to fix it. I think the problem is not that these errors occur but that the description is cryptic and doesn't point you in the right direction. An error like: "Can't process request: max_dbs_open limit reached." would be helpful.
I'm using the 1.x.x version.

[wohali]

@nintra sadly, 1.x is no longer supported. If you find the error message in CouchDB 2.x misleading/unwelcoming, let us know.