[Jenkins] view monitor failure in couchdb_views_tests/couchdb_1283

[wohali]

Expected & Current Behaviour

Normally, couchdb_views_tests's couchdb_1283 test passes. Today, it failed in Jenkins, master branch, Ubuntu 14.04, default Erlang, logs uploaded as _id jenkins-couchdb-3-2017-05-30T21:10:32.202064.

  View group shutdown
    couchdb_views_tests:315: couchdb_1283...*failed*
in function gen_server:call/2 (gen_server.erl, line 180)
in call from couchdb_views_tests:'-couchdb_1283/0-fun-22-'/0 (test/couchdb_views_tests.erl, line 368)
**exit:{normal,{gen_server,call,[<0.17099.1>,compact]}}

What I see in the couch.log is pid 0.17099.1 closing immediately after creation:

[info] 2017-05-30T21:10:27.825327Z nonode@nohost <0.17031.1> -------- Apache CouchDB has started on http://127.0.0.1:39367/
[notice] 2017-05-30T21:10:27.825734Z nonode@nohost <0.16878.1> -------- config: [couchdb] max_dbs_open set to 3 for reason nil
[info] 2017-05-30T21:10:27.825761Z nonode@nohost <0.7.0> -------- Application couch started on node nonode@nohost
[notice] 2017-05-30T21:10:27.826162Z nonode@nohost <0.16878.1> -------- config: [couchdb] delayed_commits set to false for reason nil
[info] 2017-05-30T21:10:27.931491Z nonode@nohost <0.17099.1> -------- Opening index for db: eunit-test-db-1496178627826424 idx: _design/foo sig: "0963a19eb3ef007218f1e11f0aefa2d9"
[info] 2017-05-30T21:10:27.934223Z nonode@nohost <0.17102.1> -------- Starting index update for db: eunit-test-db-1496178627826424 idx: _design/foo
[info] 2017-05-30T21:10:27.987657Z nonode@nohost <0.17102.1> -------- Index update finished for db: eunit-test-db-1496178627826424 idx: _design/foo
[notice] 2017-05-30T21:10:27.987993Z nonode@nohost <0.17050.1> -------- 127.0.0.1 - - GET /eunit-test-db-1496178627826424/_design/foo/_view/foo 200
[info] 2017-05-30T21:10:28.020920Z nonode@nohost <0.17099.1> -------- Index shutdown by monitor notice for db: eunit-test-db-1496178627826424 idx: _design/foo
[info] 2017-05-30T21:10:28.026515Z nonode@nohost <0.17099.1> -------- Closing index for db: eunit-test-db-1496178627826424 idx: _design/foo sig: "0963a19eb3ef007218f1e11f0aefa2d9" because normal

...and then the rest of the test which attempts to proceed.

Possible Solution

I wonder if this is related to #548 ?

[davisp]

If @eiri's patch for #548 doesn't fix this then I think the answer is to insert a wait for the index pid to exit due to its database being kicked out of the LRU. I'd monitor the db and index pids, wait for db exit, wait for index exit, then proceed.

I'd acquire monitors here:

https://github.com/apache/couchdb/blob/master/src/couch/test/couchdb_views_tests.erl#L345

And then wait for the 'DOWN' signal here:

https://github.com/apache/couchdb/blob/master/src/couch/test/couchdb_views_tests.erl#L352

[janl]

@davisp good call, since the compaction test below uses the same pattern.

I’ve given this a shot here: #568

[wohali]

@janl Seeing a failure since your checkin here, in the EUnit compaction daemon test:

https://travis-ci.org/apache/couchdb/jobs/243780116

module 'couchdb_compaction_daemon_tests'
  Compaction daemon tests
    couchdb_compaction_daemon_tests:74: should_compact_by_default_rule...*failed*
in function couchdb_compaction_daemon_tests:'-should_compact_by_default_rule/1-fun-6-'/1 (test/couchdb_compaction_daemon_tests.erl, line 104)
in call from couchdb_compaction_daemon_tests:'-should_compact_by_default_rule/1-fun-7-'/1 (test/couchdb_compaction_daemon_tests.erl, line 104)
**error:{assertion_failed,[{module,couchdb_compaction_daemon_tests},
                   {line,104},
                   {expression,"is_idle ( DbName )"},
                   {expected,true},
                   {value,false}]}

And again in https://travis-ci.org/apache/couchdb/jobs/243544740
And again in https://travis-ci.org/apache/couchdb/jobs/243303511

[davisp]

Gonna close this as a duplicate of #655. The previous comment from @wohali is referencing an entirely different test so am assuming that was just having too many issue tabs open at once.

[wohali]

The previous comment from me was referencing an entirely different test, but one that I bisected back to @janl 's change.

We can close for now but if that assertion comes up again we will have this "paper trail" to possibly indicate where it came from.

[davisp]

Hrm, I'm gonna open a new ticket with your comment then. I don't think these two issues are related but those failures do look to be flaky tests.