COMPASS-2110: Investigate timeouts on Travis

[pzrq]

Checklist

• What are the issues? Use 40 jobs pattern from Testing before and after COMPASS-635: "renders the home screen" functional test fix #850
  • Investigate 60000ms timeouts, such as https://travis-ci.com/10gen/compass/jobs/91799866
• Run on local macOS
  • Try DEBUG=* ?
  • IIRC - Dig up spectron-version of the above DEBUG=* ...
• Run on local Ubuntu VM
• Maybe run on local Red Hat VM?
• Other ideas as to why these are timing out (or other issues present) are welcome 👍

[pzrq]

Aggregated and sorted alphabetically to make any patterns clearer - first failure results from these initial 40 noop runs:

  1) #collections when creating & deleting collections "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[collections affected] https://travis-ci.com/10gen/compass/jobs/92374414

1) #data-service when using data-service directly "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[data-service affected] https://travis-ci.com/10gen/compass/jobs/92372412
[data-service, explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374869

 1) #explain "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92373939
[explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374311
[explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374898

  1) #indexes "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374312
[indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374313

 1) #query-bar "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92372411
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92373808
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92373811
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374407

  1) #schema "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374599
[schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374897

  1) #sidebar "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374595
[sidebar affected] https://travis-ci.com/10gen/compass/jobs/92374870

[pzrq]

Whatever is causing these test timeouts or hangs, I read as the test runner can recover from the #collections and #data-service versions, but appears to always get really confused with #explain, #indexes, #query-bar, #schema and #sidebar versions, suggesting there will be some kind of common resource shared between those tests which is implicated in the underlying error.

[pzrq]

Another potentially interesting data point is that #databases is not present at all in this data set, suggesting that anything present there is robust and works.

[pzrq]

Running on macOS I'm also seeing:

[pzrq]

Next 44 runs results:

  1) #connect "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[connect affected] https://travis-ci.com/10gen/compass/jobs/92383744

  1) #collections when creating & deleting collections "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[collections, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382796

  1) #databases "before all" hook:
     TypeError: Cannot read property 'client' of undefined

[databases, explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92384608

  1) #explain "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382603
[explain, indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382795

1) #indexes "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92383741
[indexes, query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92385060

  1) #query-bar "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382696
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382686
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92383164
[query-bar, schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92383166

  1) #schema "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[schema, sidebar affected] https://travis-ci.com/10gen/compass/jobs/92382800

  1) #schema when applying a filter in the schema tab shows a schema on refresh:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[schema affected, 5 failures] https://travis-ci.com/10gen/compass/jobs/92383616
[schema (5 failures), sidebar (2 failures)] https://travis-ci.com/10gen/compass/jobs/92384607

  1) #sidebar "before all" hook:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

[sidebar affected] https://travis-ci.com/10gen/compass/jobs/92384429

  1) #sidebar "before all" hook:
     TypeError: Cannot read property 'client' of undefined

[sidebar affected] https://travis-ci.com/10gen/compass/jobs/92384423

[pzrq]

From this data, I'm currently thinking:

• data-service code is reliable, remove logging statements so far
• extra dropDatabase had no effect (i.e. keep the removal) 40d54e4
• 1) #databases "before all" hook: TypeError: Cannot read property 'client' of undefined can be part of what I'm calling the main failure chain: https://travis-ci.com/10gen/compass/jobs/92384608
• from #sidebar the main failure chain may or may not be triggered by two separate issues:
  • 1) #sidebar "before all" hook: Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
  • 1) #sidebar "before all" hook: TypeError: Cannot read property 'client' of undefined
• there appear to be at least three other lower occurrence issues lurking:
  • 1) #connect "before all" hook: Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.
  • 1) #collections when creating & deleting collections "before all" hook: (which can reset successfully, and also later trigger the main failure chain)
  • 1) #schema when applying a filter in the schema tab shows a schema on refresh, with and without the main failure chain

[pzrq]

Results from the next 44 runs (Friday evening + Tue morning AEDT):

(Friday only so far)

  1) #query-bar when using advanced query options when applying queries from the documents tab when applying a projection returns only the fields included in the project plus _id:
      AssertionError: expected [ '_id', 'member_id' ] to deeply equal [ '_id', 'member_id', 'name' ]
      + expected - actual
       [
         "_id"
         "member_id"
      +  "name"
       ]

https://travis-ci.com/10gen/compass/jobs/92580507

(Friday only so far)

  1) #launch "before all" hook:
     TypeError: Cannot read property 'client' of undefined
      at test/functional/_launch.test.js:12:27
      at process._tickDomainCallback (internal/process/next_tick.js:129:7)

https://travis-ci.com/10gen/compass/jobs/92582209
https://travis-ci.com/10gen/compass/jobs/92582391
https://travis-ci.com/10gen/compass/jobs/92584116
https://travis-ci.com/10gen/compass/jobs/92584117
https://travis-ci.com/10gen/compass/jobs/92584107

(Tuesday only so far)

  1) #schema when applying a filter in the schema tab shows a schema on refresh:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

https://travis-ci.com/10gen/compass/jobs/92919029
https://travis-ci.com/10gen/compass/jobs/92921157

[pzrq]

Plan for today

As the schema on fresh test appears to be present on both the container-based infrastructure and on the original infrastructure going to target it.

  1) #schema when applying a filter in the schema tab shows a schema on refresh:
     Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

Will use any verification Travis run if I make progress on this schema on refresh to see if the #launch or name issues come back.

[pzrq]

So far this is not a solution for shows a schema on refresh:
https://travis-ci.com/10gen/compass/jobs/92927263

[pzrq]

Well if https://travis-ci.com/10gen/compass/jobs/92927263 was the only failure in >80 jobs I think that's more than enough to call this done, follow up work to get this onto master will be in https://jira.mongodb.org/browse/COMPASS-2159