Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error opening views: badmatch/ badarg, ets member mem3_openers #1106

Closed
jvegaseg opened this issue Jan 13, 2018 · 13 comments
Closed

Error opening views: badmatch/ badarg, ets member mem3_openers #1106

jvegaseg opened this issue Jan 13, 2018 · 13 comments
Labels
bug need more info waiting on user

Comments

@jvegaseg
Copy link

jvegaseg commented Jan 13, 2018 •
edited

We have a 6-node cluster configured with q=16.

Every time we restart the server, couchdb logs same errors on startup.

Expected Behavior

It should start without errors

Current Behavior

It starts with errors:



[info] 2018-01-13T17:07:55.495800Z couchdb@X.X.X.X <0.7.0> -------- Application couch_log started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.500099Z couchdb@X.X.X.X <0.7.0> -------- Application folsom started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.535214Z couchdb@X.X.X.X <0.7.0> -------- Application couch_stats started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.535340Z couchdb@X.X.X.X <0.7.0> -------- Application khash started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.542427Z couchdb@X.X.X.X <0.7.0> -------- Application couch_event started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.547405Z couchdb@X.X.X.X <0.7.0> -------- Application ibrowse started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.551422Z couchdb@X.X.X.X <0.7.0> -------- Application ioq started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.551568Z couchdb@X.X.X.X <0.7.0> -------- Application mochiweb started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.557754Z couchdb@X.X.X.X <0.213.0> -------- Apache CouchDB 2.1.1 is starting.

[info] 2018-01-13T17:07:55.557893Z couchdb@X.X.X.X <0.214.0> -------- Starting couch_sup
[info] 2018-01-13T17:07:55.695716Z couchdb@X.X.X.X <0.213.0> -------- Apache CouchDB has started. Time to relax.

[info] 2018-01-13T17:07:55.695890Z couchdb@X.X.X.X <0.213.0> -------- Apache CouchDB has started on http://0.0.0.0:5986/
[info] 2018-01-13T17:07:55.696003Z couchdb@X.X.X.X <0.7.0> -------- Application couch started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.696139Z couchdb@X.X.X.X <0.7.0> -------- Application ets_lru started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.720481Z couchdb@X.X.X.X <0.7.0> -------- Application rexi started on node 'couchdb@X.X.X.X'
[error] 2018-01-13T17:07:55.729736Z couchdb@X.X.X.X emulator -------- Error in process <0.323.0> on node 'couchdb@X.X.X.X' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-01-13T17:07:55.729874Z couchdb@X.X.X.X <0.249.0> -------- Error opening view group `_auth` from database `shards/50000000-5fffffff/_users.1510390507`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[info] 2018-01-13T17:07:55.784275Z couchdb@X.X.X.X <0.365.0> -------- Starting compaction for db "shards/30000000-3fffffff/_global_changes.1510390507"
[info] 2018-01-13T17:07:55.786450Z couchdb@X.X.X.X <0.7.0> -------- Application mem3 started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.786533Z couchdb@X.X.X.X <0.7.0> -------- Application fabric started on node 'couchdb@X.X.X.X'
[notice] 2018-01-13T17:07:55.851770Z couchdb@X.X.X.X <0.365.0> -------- Compaction swap for db: ./data/shards/30000000-3fffffff/_global_changes.1510390507.couch 7905486 53448
[info] 2018-01-13T17:07:55.852253Z couchdb@X.X.X.X <0.365.0> -------- Compaction for db "shards/30000000-3fffffff/_global_changes.1510390507" completed.
[info] 2018-01-13T17:07:55.905176Z couchdb@X.X.X.X <0.7.0> -------- Application chttpd started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.917494Z couchdb@X.X.X.X <0.7.0> -------- Application couch_index started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.917531Z couchdb@X.X.X.X <0.7.0> -------- Application couch_mrview started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.917616Z couchdb@X.X.X.X <0.7.0> -------- Application couch_plugins started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:55.917616Z couchdb@X.X.X.X <0.7.0> -------- Application couch_plugins started on node 'couchdb@X.X.X.X'
[notice] 2018-01-13T17:07:55.968574Z couchdb@X.X.X.X <0.93.0> -------- config: [features] scheduler set to true for reason nil
[info] 2018-01-13T17:07:55.993761Z couchdb@X.X.X.X <0.1278.0> -------- Opening index for db: _users idx: _design/_auth sig: "3e823c2a4383ac0c18d4e574135a5b08"
[info] 2018-01-13T17:07:56.004620Z couchdb@X.X.X.X <0.1302.0> -------- Opening index for db: _replicator idx: _design/_replicator sig: "3e823c2a4383ac0c18d4e574135a5b08"
[info] 2018-01-13T17:07:56.013595Z couchdb@X.X.X.X <0.7.0> -------- Application couch_replicator started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.019517Z couchdb@X.X.X.X <0.7.0> -------- Application couch_peruser started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.025704Z couchdb@X.X.X.X <0.7.0> -------- Application ddoc_cache started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.036404Z couchdb@X.X.X.X <0.7.0> -------- Application global_changes started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.036492Z couchdb@X.X.X.X <0.7.0> -------- Application jiffy started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.040751Z couchdb@X.X.X.X <0.7.0> -------- Application mango started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.045152Z couchdb@X.X.X.X <0.7.0> -------- Application setup started on node 'couchdb@X.X.X.X'
[info] 2018-01-13T17:07:56.045228Z couchdb@X.X.X.X <0.7.0> -------- Application snappy started on node 'couchdb@X.X.X.X'

Possible Solution

I do not know-

Configuration

We use default settings except:

local.ini

[cluster]
q = 16

[log]
level = warning

vm.args

-name couchdb@X.X.X.120
-setcookie XXXXXXXXXXXXXX
-kernel inet_dist_listen_min 9100
-kernel inet_dist_listen_max 9200

-kernel error_logger silent
-sasl sasl_error_logger false

+K true
+A 16
+Bd -noinput

sys.config

[
    {lager, [
        {error_logger_hwm, 1000},
        {error_logger_redirect, true},
        {handlers, [
            {lager_console_backend, [debug, {
                lager_default_formatter,
                [
                    date, " ", time,
                    " [", severity, "] ",
                    node, " ", pid, " ",
                    message,
                    "\n"
                ]
            }]}
        ]},
        {inet_dist_listen_min, 9100},
        {inet_dist_listen_max, 9200}
    ]}
].

Environment

Software is running on virtualized Centos 7 using OpenVZ 7.
Each node has same configuration.
Each node has 16 cores.
@wohali
Copy link
Member

wohali commented Jan 16, 2018

Do you get the exact same error on every node, or does the shard in the error line change?

@jvegaseg
Copy link
Author

jvegaseg commented Jan 17, 2018 •
edited

It is the exact same shard when error appears in other nodes.

@jvegaseg
Copy link
Author

How can I check if everything is OK with the shard (shards/50000000-5fffffff/_users.1510390507)?
Error is reported only when we restart couchdb

@wohali
Copy link
Member

wohali commented Feb 1, 2018

Try replicating the _users database to another database.

If this works, you could try scheduling a maintenance window, deleting _users, recreating _users and replicating all of the content back to your _users database.

@janl janl closed this as completed Feb 28, 2018
@tarasvaskiv
Copy link

tarasvaskiv commented Mar 25, 2018 •
edited by wohali

I also having the similar errors couchdb2.1

/home/t/Projects/x_b_11/couchdb/bin/couchdb -couch_ini /home/t/Projects/x_b_11/etc/couchdb.ini
[error] 2018-03-25T09:40:38.351238Z couchdb@127.0.0.1 emulator -------- Error in process <0.258.0> on node 'couchdb@127.0.0.1' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"_replicator">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-03-25T09:40:38.351315Z couchdb@127.0.0.1 <0.237.0> -------- Error opening view group `_replicator` from database `shards/00000000-ffffffff/_replicator.1521898014`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_replicator">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}
[error] 2018-03-25T09:40:38.354010Z couchdb@127.0.0.1 emulator -------- Error in process <0.276.0> on node 'couchdb@127.0.0.1' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-03-25T09:40:38.354104Z couchdb@127.0.0.1 <0.237.0> -------- Error opening view group `_auth` from database `shards/00000000-ffffffff/_users.1521970759`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,486}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,389}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,378}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,407}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,96}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,332}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,253}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,227}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',3,[{file,"src/couch_compaction_daemon.erl"},{line,141}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,278}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,185}]},{filelib,do_fold_files2,8,[{file,"filelib.erl"},{line,195}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,267}]}]}}

This happens every time I start couchdb with command /home/t/Projects/x_b_11/couchdb/bin/couchdb -couch_ini /home/t/Projects/x_b_11/etc/couchdb.ini Seems to me couchdb2.1 is not ready for production yet. To many errors.

@wohali wohali reopened this Mar 25, 2018
@wohali
Copy link
Member

wohali commented Mar 25, 2018

@tarasvaskiv It's definitely ready for production - it's in use in tens of thousands of installations world wide that I know of, personally. And I'm just one person.

We do need to fix this problem, though. Reopening.

@wohali wohali changed the title CouchDB 2.1.1: errors on startup "Error opening view group _auth" CouchDB 2.1.1: error opening views: badmatch/ badarg, ets member mem3_openers [_users|_auth] Mar 25, 2018
@wohali wohali added this to the 2.2.0 milestone Mar 25, 2018
@ThibaultJanBeyer
Copy link

ThibaultJanBeyer commented Apr 3, 2018 •
edited

Got a similar issue

error | "unknown_error"
reason | "badarg"
ref | 749894463

It happened when I changed the settings values of the auth timeout. I Set it to 86.400 for one day. But I think it took 86 seconds because of the .. So I tried to set it to 86400 but got the error mentioned above asap as the value "saved"…

Please fix.

For the meantime, is there any way to undo the changes?

@jareware jareware mentioned this issue Apr 11, 2018
Closed
@wohali
Copy link
Member

wohali commented Apr 18, 2018

@ThibaultJanBeyer Not fixing this issue, but you can revert that change by editing your local.ini (or the last ini file in your local.d directory and removing the timeout change, then restarting couchdb.

@wohali wohali changed the title CouchDB 2.1.1: error opening views: badmatch/ badarg, ets member mem3_openers [_users|_auth] CouchDB 2.1.1: error opening views: badmatch/ badarg, ets member mem3_openers Apr 18, 2018
@wohali wohali changed the title CouchDB 2.1.1: error opening views: badmatch/ badarg, ets member mem3_openers Error opening views: badmatch/ badarg, ets member mem3_openers Apr 18, 2018
@ThibaultJanBeyer
Copy link

Yes, that did resolve the problem. However, generally wrong values should not be permitted to enter in the UI or at least not cause a total crash. Thanks for the Help!

@davedoesdev
Copy link

Seeing this too:

[error] 2018-05-18T20:05:47.597777Z couchdb@127.0.0.1 emulator -------- Error in process <0.288.0> on node 'couchdb@127.0.0.1' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,475}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,380}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,369}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,398}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-05-18T20:05:47.597860Z couchdb@127.0.0.1 <0.273.0> -------- Error opening view group `_auth` from database `shards/40000000-5fffffff/_users.1526504179`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"_users">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,475}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,380}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,369}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,398}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,334}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,257}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,228}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',4,[{file,"src/couch_compaction_daemon.erl"},{line,142}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,312}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,107}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,112}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,302}]}]}}

Doesn't seem to affect functionality though.

@janl
Copy link
Member

janl commented Jul 8, 2018

@davedoesdev what q value do you have configured?

@janl
Copy link
Member

janl commented Jul 9, 2018

reflecting on this: we’d need a lot more info on this in terms of system setup, cluster config, load on couch, potential other load on machines, network etc. I’ll close this until we have a report with a more complete set of reproducible steps.

@janl janl closed this as completed Jul 9, 2018
@janl janl removed this from the 2.2.0 milestone Jul 9, 2018
@mariocasciaro
Copy link

Trying to add some more information here since I have similar errors in my log (actually a lot of them):

[error] 2018-11-12T15:03:04.488080Z couchdb@xxx.xxx emulator -------- Error in process <0.1582.0> on node 'couchdb@xxx.xxx' with exit value:
{badarg,[{ets,member,[mem3_openers,<<"xxx">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}

[error] 2018-11-12T15:03:04.488130Z couchdb@xxx.xxx <0.244.0> -------- Error opening view group `rel` from database `shards/00000000-ffffffff/xxx.1534776614`: {'EXIT',{{badmatch,{badarg,[{ets,member,[mem3_openers,<<"xxx">>],[]},{mem3_shards,maybe_spawn_shard_writer,3,[{file,"src/mem3_shards.erl"},{line,476}]},{mem3_shards,load_shards_from_db,2,[{file,"src/mem3_shards.erl"},{line,381}]},{mem3_shards,load_shards_from_disk,1,[{file,"src/mem3_shards.erl"},{line,370}]},{mem3_shards,load_shards_from_disk,2,[{file,"src/mem3_shards.erl"},{line,399}]},{mem3_shards,for_docid,3,[{file,"src/mem3_shards.erl"},{line,91}]},{fabric_doc_open,go,3,[{file,"src/fabric_doc_open.erl"},{line,38}]},{couch_index_server,'-get_index/3-fun-0-',2,[{file,"src/couch_index_server.erl"},{line,68}]}]}},[{couch_mrview,get_info,2,[{file,"src/couch_mrview.erl"},{line,337}]},{couch_compaction_daemon,maybe_compact_view,3,[{file,"src/couch_compaction_daemon.erl"},{line,257}]},{couch_compaction_daemon,maybe_compact_views,3,[{file,"src/couch_compaction_daemon.erl"},{line,228}]},{couch_compaction_daemon,'-compact_loop/1-fun-0-',4,[{file,"src/couch_compaction_daemon.erl"},{line,142}]},{couch_server,'-all_databases/2-fun-0-',4,[{file,"src/couch_server.erl"},{line,315}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,108}]},{couch_util,fold_files2,6,[{file,"src/couch_util.erl"},{line,113}]},{couch_server,all_databases,2,[{file,"src/couch_server.erl"},{line,305}]}]}}

Usually those errors are preceded by lines like these:

[notice] 2018-11-12T15:02:58.436921Z couchdb@xxx.xxx <0.276.0> -------- rexi_server_mon : cluster unstable
[notice] 2018-11-12T15:02:58.436962Z couchdb@xxx.xxx <0.279.0> -------- rexi_server_mon : cluster unstable
[notice] 2018-11-12T15:02:58.437193Z couchdb@xxx.xxx <0.275.0> -------- rexi_server : cluster unstable
[notice] 2018-11-12T15:02:58.437287Z couchdb@xxx.xxx <0.278.0> -------- rexi_buffer : cluster unstable
[notice] 2018-11-12T15:02:58.469005Z couchdb@xxx.xxx <0.276.0> -------- rexi_server_mon : cluster unstable
[notice] 2018-11-12T15:02:58.469046Z couchdb@xxx.xxx <0.279.0> -------- rexi_server_mon : cluster unstable
[notice] 2018-11-12T15:02:58.469310Z couchdb@xxx.xxx <0.275.0> -------- rexi_server : cluster unstable
[notice] 2018-11-12T15:02:58.469357Z couchdb@xxx.xxx <0.278.0> -------- rexi_buffer : cluster unstable

I also noticed that the stat number of rexi_DOWN messages handled is very high, it increases of about 1000 every 10 minutes.

Other info:

q=1
n=3

Databases: around 200, most of them < 1mb, a few up to 40mb, total size 200mb
Load on the database: max 100 requests per minute
Load on machines: very low, CPU avg 2%, Max memory usage 40%
Machines specs: 1 vCpu, 2GB ram
I can consistently see those errors on all the 3 machines in the cluster and all have about the same number of 'rexi_DOWN messages handled'.

The same application and database configuration (but with different data) is running on another environment, actually with higher load and many more databases and I can't see all those error messages.

davedoesdev added a commit to davedoesdev/pub-keystore that referenced this issue Mar 27, 2020
@davedoesdev
Work around apache/couchdb#1106 in tests
1009c43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug need more info waiting on user
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@janl @mariocasciaro @wohali @davedoesdev @ThibaultJanBeyer @jvegaseg @tarasvaskiv