Repeated mango test failures

[wohali]

We're getting a lot of transient Mango test errors on PRs. I've gotten 4-5 over the past day on just 2 PRs.

Here's the output of one:
https://gist.github.com/wohali/86670a48fe20c488bee636c29e9c4ddb

The logs are all uploaded for runs today, you should be able to use the _all_docs view on the log DB to see the rest.

@tonysun83 @garrensmith Can we get these tests stabilized please? Thanks!

[wohali]

I didn't look at each one extensively, but the last 4-5 of these for sure should only be Mango failures:

{"id":"travis-couchdb-292792838-2017-10-25T20:10:41.928582","key":"travis-couchdb-292792838-2017-10-25T20:10:41.928582","value":{"rev":"2-bbacce13c65b066364140ed9e3d5f4ca"}},
{"id":"travis-couchdb-292798810-2017-10-25T20:35:38.862808","key":"travis-couchdb-292798810-2017-10-25T20:35:38.862808","value":{"rev":"2-40d635cb95135050218862cf5ee64774"}},
{"id":"travis-couchdb-292798815-2017-10-25T20:39:14.149481","key":"travis-couchdb-292798815-2017-10-25T20:39:14.149481","value":{"rev":"2-c154555432f6409f9ed48556604114f5"}},
{"id":"travis-couchdb-292809826-2017-10-25T21:16:18.790702","key":"travis-couchdb-292809826-2017-10-25T21:16:18.790702","value":{"rev":"2-6d9f7e13f5546f727858cd1f2df65d37"}},
{"id":"travis-couchdb-292851785-2017-10-25T22:06:51.753064","key":"travis-couchdb-292851785-2017-10-25T22:06:51.753064","value":{"rev":"2-b00b3446ac830e402e8c89a684c6c876"}},
{"id":"travis-couchdb-292857395-2017-10-25T22:37:53.761593","key":"travis-couchdb-292857395-2017-10-25T22:37:53.761593","value":{"rev":"2-64e37c005a949bac36527c370aaba3d2"}},
{"id":"travis-couchdb-292857399-2017-10-25T22:46:54.115940","key":"travis-couchdb-292857399-2017-10-25T22:46:54.115940","value":{"rev":"2-e2c9dda6cebb9c47f3ca9af304fa8ec2"}},
{"id":"travis-couchdb-292857399-2017-10-25T23:11:54.824570","key":"travis-couchdb-292857399-2017-10-25T23:11:54.824570","value":{"rev":"2-1112062ba0cc8f171a4568c3d24770db"}},
{"id":"travis-couchdb-292857399-2017-10-25T23:41:04.643705","key":"travis-couchdb-292857399-2017-10-25T23:41:04.643705","value":{"rev":"2-201f8e11d385820b891f148179902493"}}

[wohali]

Caught another, with makefile log this time:

.............FSSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS..........

======================================================================

FAIL: test_recreate_index (01-index-crud-test.IndexCrudTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 204, in test_recreate_index

    self.assertEqual(pre_indexes, post_indexes)

AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp...

Second list contains 1 additional elements.

First extra element 1:

{u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e', u'type': u'json', u'name': u'idx_recreate', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}}

  [{u'ddoc': None,

    u'def': {u'fields': [{u'_id': u'asc'}]},

    u'name': u'_all_docs',

-   u'type': u'special'}]

?                       ^

+   u'type': u'special'},

?                       ^

+  {u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e',

+   u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}},

+   u'name': u'idx_recreate',

+   u'type': u'json'}]

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 404 58

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_79712e850ec24bd9a56dc85e37169ec8?q=1&n=1 HTTP/1.1" 201 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------

Ran 273 tests in 135.130s

FAILED (SKIP=131, failures=1)

make[1]: *** [mango-test] Error 1

make[1]: Leaving directory `/home/travis/build/apache/couchdb'

make: *** [check] Error 2

The command "make check" exited with 2.
after_failure

1.40s$ build-aux/logfile-uploader.py
Uploading logfiles...

https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-292905867-2017-10-26T00:47:21.703579/couchlog.tar.gz

{"ok":true,"id":"travis-couchdb-292905867-2017-10-26T00:47:21.703579","rev":"2-2dbfa0eec19310bc919a21e7130c1bb7"}

[wohali]

Same error again:

.......F......SSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS..........

======================================================================

FAIL: test_delete_idx_escaped (01-index-crud-test.IndexCrudTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 136, in test_delete_idx_escaped

    self.assertEqual(pre_indexes, post_indexes)

AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp...

First differing element 1:

{u'ddoc': u'_design/999f65adef69ebbcfd1134e21f91bba09e58997e', u'type': u'json', u'name': u'idx_01', u'def': {u'fields': [{u'foo': u'asc'}, {u'bar': u'asc'}], u'partial_filter_selector': {}}}

{u'ddoc': u'_design/0434d2e3399f8af08555d4022b187ee714cb35ca', u'type': u'json', u'name': u'idx_del_1', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}}

Second list contains 1 additional elements.

First extra element 2:

{u'ddoc': u'_design/999f65adef69ebbcfd1134e21f91bba09e58997e', u'type': u'json', u'name': u'idx_01', u'def': {u'fields': [{u'foo': u'asc'}, {u'bar': u'asc'}], u'partial_filter_selector': {}}}

  [{u'ddoc': None,

    u'def': {u'fields': [{u'_id': u'asc'}]},

    u'name': u'_all_docs',

    u'type': u'special'},

+  {u'ddoc': u'_design/0434d2e3399f8af08555d4022b187ee714cb35ca',

+   u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}},

+   u'name': u'idx_del_1',

+   u'type': u'json'},

   {u'ddoc': u'_design/999f65adef69ebbcfd1134e21f91bba09e58997e',

    u'def': {u'fields': [{u'foo': u'asc'}, {u'bar': u'asc'}],

             u'partial_filter_selector': {}},

    u'name': u'idx_01',

    u'type': u'json'}]

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_e65da969f33c4da3a1aa962e7cfdf007 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_e65da969f33c4da3a1aa962e7cfdf007 HTTP/1.1" 404 58

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_e65da969f33c4da3a1aa962e7cfdf007?q=1&n=1 HTTP/1.1" 201 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_e65da969f33c4da3a1aa962e7cfdf007/_index HTTP/1.1" 200 93

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_e65da969f33c4da3a1aa962e7cfdf007/_index?; HTTP/1.1" 200 278

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_e65da969f33c4da3a1aa962e7cfdf007/_index HTTP/1.1" 200 96

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_e65da969f33c4da3a1aa962e7cfdf007/_index?; HTTP/1.1" 200 278

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_e65da969f33c4da3a1aa962e7cfdf007/_index?; HTTP/1.1" 200 434

--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------

Ran 273 tests in 137.926s

FAILED (SKIP=131, failures=1)

make[1]: *** [mango-test] Error 1

make[1]: Leaving directory `/home/travis/build/apache/couchdb'

make: *** [check] Error 2

Logfile are at https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-292933918-2017-10-26T02:33:43.620626/couchlog.tar.gz

And a different error:

.............FSSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS..........

======================================================================

FAIL: test_recreate_index (01-index-crud-test.IndexCrudTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 204, in test_recreate_index

    self.assertEqual(pre_indexes, post_indexes)

AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp...

Second list contains 1 additional elements.

First extra element 1:

{u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e', u'type': u'json', u'name': u'idx_recreate', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}}

  [{u'ddoc': None,

    u'def': {u'fields': [{u'_id': u'asc'}]},

    u'name': u'_all_docs',

-   u'type': u'special'}]

?                       ^

+   u'type': u'special'},

?                       ^

+  {u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e',

+   u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}},

+   u'name': u'idx_recreate',

+   u'type': u'json'}]

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_b4a828a268764356b8b03ba76a2617b3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3 HTTP/1.1" 404 58

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_b4a828a268764356b8b03ba76a2617b3?q=1&n=1 HTTP/1.1" 201 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_b4a828a268764356b8b03ba76a2617b3/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_b4a828a268764356b8b03ba76a2617b3/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_b4a828a268764356b8b03ba76a2617b3/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_b4a828a268764356b8b03ba76a2617b3/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_b4a828a268764356b8b03ba76a2617b3/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_b4a828a268764356b8b03ba76a2617b3/_index?; HTTP/1.1" 200 271

--------------------- >> end captured logging << ---------------------

This one is at https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-292933922-2017-10-26T02:34:33.449311/couchlog.tar.gz

[wohali]

And another:

.............FSSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS..........

======================================================================

FAIL: test_recreate_index (01-index-crud-test.IndexCrudTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 204, in test_recreate_index

    self.assertEqual(pre_indexes, post_indexes)

AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp...

Second list contains 1 additional elements.

First extra element 1:

{u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e', u'type': u'json', u'name': u'idx_recreate', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}}

  [{u'ddoc': None,

    u'def': {u'fields': [{u'_id': u'asc'}]},

    u'name': u'_all_docs',

-   u'type': u'special'}]

?                       ^

+   u'type': u'special'},

?                       ^

+  {u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e',

+   u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}},

+   u'name': u'idx_recreate',

+   u'type': u'json'}]

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 404 58

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_79712e850ec24bd9a56dc85e37169ec8?q=1&n=1 HTTP/1.1" 201 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271
--------------------- >> end captured logging << ---------------------

Logs at https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-292905867-2017-10-26T00:47:21.703579

[garrensmith]

I think I know what the issue is. I can take a look today All misspelling thanks to my iPhone.

…

________________________________ From: Joan Touzet <notifications@github.com> Sent: Thursday, October 26, 2017 5:34:50 AM To: apache/couchdb Cc: garren smith; Mention Subject: Re: [apache/couchdb] Repeated mango test failures (#918) And another: .............FSSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS.......... ====================================================================== FAIL: test_recreate_index (01-index-crud-test.IndexCrudTests)

---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 204, in test_recreate_index self.assertEqual(pre_indexes, post_indexes) AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp... Second list contains 1 additional elements. First extra element 1: {u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e', u'type': u'json', u'name': u'idx_recreate', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}} [{u'ddoc': None, u'def': {u'fields': [{u'_id': u'asc'}]}, u'name': u'_all_docs', - u'type': u'special'}] ? ^ + u'type': u'special'}, ? ^ + {u'ddoc': u'_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e', + u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}, + u'name': u'idx_recreate', + u'type': u'json'}]

-------------------- >> begin captured logging << -------------------- urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 200 12 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8 HTTP/1.1" 404 58 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_79712e850ec24bd9a56dc85e37169ec8?q=1&n=1 HTTP/1.1" 201 12 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 112 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index HTTP/1.1" 200 99 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index/_design/833d6679437cfc03e8b3c6a5f5a76c2637675c1e/json/idx_recreate?w=3 HTTP/1.1" 200 12 urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_79712e850ec24bd9a56dc85e37169ec8/_index?; HTTP/1.1" 200 271 --------------------- >> end captured logging << --------------------- Logs at https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-292905867-2017-10-26T00:47:21.703579 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#918 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAK9Algk5rWMlrsQXF-gaPb6geDnLyEVks5sv_3agaJpZM4QG0-t>.

[tonysun83]

@wohali : Even though it's a non-deterministic bug, I'm still trying to recreate in my env locally. Looking at the log you posted, the job was:
https://travis-ci.org/apache/couchdb/jobs/292905867
I saw that the branch was:
erlang-20.1
But it seems like the travis builds passed. Is there a specific branch that recently failed that you can point me to that I can checkout in hopes of recreating? I know HOW the test is failing...now need to recreate and figure out WHY.

[wohali]

@tonysun83 Sorry about the failures. I have to re-run those builds to get PRs merged now that we enforce travis passing to merge a PR. That's why I'm pasting the output here.

Here's another failure. It was on a PR change only to Makefile that affects the release process, so definitely not any change to Mango, and in fact a change that has zero impact on the CI workflow (since it's just to the make dist target.) Given this, you should be able to reproduce the error on master.

It's on PR #922 for reference, and Erlang 18.3. (I've seen the error across all Erlang versions.)

........F.....SSS................................................................SSSSSSSSSSSSSSSSSSSSSSSSSS...........SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS....................................SS......SSSSSS..........

======================================================================

FAIL: test_delete_idx_no_design (01-index-crud-test.IndexCrudTests)

----------------------------------------------------------------------

Traceback (most recent call last):

  File "/home/travis/build/apache/couchdb/src/mango/test/01-index-crud-test.py", line 160, in test_delete_idx_no_design

    self.assertEqual(pre_indexes, post_indexes)

AssertionError: Lists differ: [{u'ddoc': None, u'type': u'sp... != [{u'ddoc': None, u'type': u'sp...

Second list contains 1 additional elements.

First extra element 1:

{u'ddoc': u'_design/cf4362f791c16fdda8f33c39c67c8a4a53e524ff', u'type': u'json', u'name': u'idx_del_3', u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}}}

  [{u'ddoc': None,

    u'def': {u'fields': [{u'_id': u'asc'}]},

    u'name': u'_all_docs',

-   u'type': u'special'}]

?                       ^

+   u'type': u'special'},

?                       ^

+  {u'ddoc': u'_design/cf4362f791c16fdda8f33c39c67c8a4a53e524ff',

+   u'def': {u'fields': [{u'bing': u'asc'}], u'partial_filter_selector': {}},

+   u'name': u'idx_del_3',

+   u'type': u'json'}]

-------------------- >> begin captured logging << --------------------

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_1b65a9866b074034b6e6a1100765bb66 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_1b65a9866b074034b6e6a1100765bb66 HTTP/1.1" 404 58

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "PUT /mango_test_1b65a9866b074034b6e6a1100765bb66?q=1&n=1 HTTP/1.1" 201 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_1b65a9866b074034b6e6a1100765bb66/_index?; HTTP/1.1" 200 112

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "POST /mango_test_1b65a9866b074034b6e6a1100765bb66/_index HTTP/1.1" 200 96

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_1b65a9866b074034b6e6a1100765bb66/_index?; HTTP/1.1" 200 268

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "DELETE /mango_test_1b65a9866b074034b6e6a1100765bb66/_index/cf4362f791c16fdda8f33c39c67c8a4a53e524ff/json/idx_del_3?w=3 HTTP/1.1" 200 12

urllib3.connectionpool: DEBUG: http://127.0.0.1:15984 "GET /mango_test_1b65a9866b074034b6e6a1100765bb66/_index?; HTTP/1.1" 200 268

--------------------- >> end captured logging << ---------------------

https://couchdb-vm2.apache.org/ci_errorlogs/travis-couchdb-293262014-2017-10-26T17:45:40.369364/couchlog.tar.gz

[wohali]

@tonysun83 OK, here's a run I don't need to re-run to merge a PR.

https://travis-ci.org/apache/couchdb/jobs/293295681

Go nuts ;)

[wohali]

@tonysun83 One more, this time on master, and it's a different failure:

https://travis-ci.org/apache/couchdb/jobs/293262014

[tonysun83]

@wohali : I made a PR: #923. My thoughts are in the overview. The last test failure is something different and related to the users db. I'll take a look tomorrow.