[CI] DocsClientYamlTestSuiteIT test {yaml=reference/ccr/getting-started/line_96}

[martijnvg]

The get remote connections snippet in ccr getting started docs fails, because the remote cluster is not connected yet. Looks like a timing issue.

This reproduce in 6.x and 6.5 regularly (random seed does not matter). On master this does not reproduce.

[2018-11-13T22:24:59,896][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "host" : "127.0.0.1:44037",
  1>     "body" : {
  1>       "leader" : {
  1>         "seeds" : [ ],
  1>         "http_addresses" : [ ],
  1>         "connected" : false,
  1>         "num_nodes_connected" : 0,
  1>         "max_connections_per_cluster" : 3,
  1>         "initial_connect_timeout" : "30s",
  1>         "skip_unavailable" : false
  1>       }
  1>     },
  1>     "master" : "nV3HRmvCSDiqBDQUOgCjgQ",
  1>     "transport_host" : "127.0.0.1:44361"
  1>   }
  1> }]
  1> [2018-11-13T22:24:59,974][INFO ][o.e.s.DocsClientYamlTestSuiteIT] [test] [yaml=reference/ccr/getting-started/line_96] after test
  1> [2018-11-13T22:24:59,974][ERROR][o.e.s.DocsClientYamlTestSuiteIT] [test] This failing test was generated by documentation starting at reference/ccr/getting-started.asciidoc:line_96. It may include many snippets. See docs/README.asciidoc for an explanation of test generation.
ERROR   0.14s | DocsClientYamlTestSuiteIT.test {yaml=reference/ccr/getting-started/line_96} <<< FAILURES!
   > Throwable #1: java.lang.RuntimeException: Failure at [reference/ccr/getting-started:45]: element was a list with 0 elements, but [0] was out of bounds
   >    at __randomizedtesting.SeedInfo.seed([F8B16B31156152AD:70E554EBBB9D3F55]:0)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:408)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:383)
   >    at java.lang.Thread.run(Thread.java:748)
   > Caused by: java.lang.IllegalArgumentException: element was a list with 0 elements, but [0] was out of bounds
   >    at org.elasticsearch.test.rest.yaml.ObjectPath.evaluate(ObjectPath.java:114)
   >    at org.elasticsearch.test.rest.yaml.ObjectPath.evaluate(ObjectPath.java:90)
   >    at org.elasticsearch.test.rest.yaml.ObjectPath.evaluate(ObjectPath.java:79)
   >    at org.elasticsearch.test.rest.yaml.Stash.unstash(Stash.java:116)
   >    at org.elasticsearch.test.rest.yaml.Stash.getValue(Stash.java:97)
   >    at org.elasticsearch.test.rest.yaml.Stash.getValue(Stash.java:182)
   >    at org.elasticsearch.test.rest.yaml.Stash.unstashObject(Stash.java:140)
   >    at org.elasticsearch.test.rest.yaml.Stash.unstashObject(Stash.java:161)
   >    at org.elasticsearch.test.rest.yaml.Stash.unstashObject(Stash.java:161)
   >    at org.elasticsearch.test.rest.yaml.Stash.replaceStashedValues(Stash.java:129)
   >    at org.elasticsearch.test.rest.yaml.section.Assertion.resolveExpectedValue(Assertion.java:53)
   >    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
   >    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:399)
   >    ... 37 more
   > Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
   >    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
   >    at java.util.ArrayList.get(ArrayList.java:433)
   >    at org.elasticsearch.test.rest.yaml.ObjectPath.evaluate(ObjectPath.java:110)
   >    ... 49 more

Build url: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+intake/176/

[elasticmachine]

Pinging @elastic/es-distributed

[ywelsch]

@martijnvg is this still an issue in 6.7?

[dnhatn]

Looks like a timing issue.

I think so. I pushed 9c33d75 to unmute this test.

[dnhatn]

This is still an issue on 6.8: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.8+multijob-unix-compatibility/os=opensuse/8/console

[ywelsch]

I've had a look here. The issue is that because remote connections are not established synchronously when registering a new remote host, the check to GET /_remote/info races connections being established. This is the same in 6.8 and 7.x/master. The difference is how the remote info is reported. On 6.8, seeds is returning an empty list when there isn't a connection to any node yet, whereas 7.x/master already reports the entry.

This means that the test response replacement // TEST[s/127.0.0.1:9300/$body.leader.seeds.0/] works for 7.x/master, but not 6.x. Unfortunately, the REST test framework does not allow to specify a more powerful pattern here. As such, I'm closing this issue as wontfix. The TESTRESPONSE is not verifying much anyway, as nearly all the fields are mocked.