[CI] MixedClusterClientYamlTestSuiteIT test {p0=search/110_field_collapsing/field collapsing and inner_hits}

[astefan]

This one doesn't reproduce for me.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2019/442/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-windows-compatibility/os=windows-2016/442/console
https://gradle-enterprise.elastic.co/s/7sb66cpgvp472
https://gradle-enterprise.elastic.co/s/ebbbgtn25jytw

java.lang.AssertionError: Failure at [search/110_field_collapsing:245]: hits.total didn't match expected value:
                    hits.total: expected Integer [6] but was Integer [5]

	at __randomizedtesting.SeedInfo.seed([65F24D0830A9A991:EDA672D29E55C469]:0)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:405)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:382)

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[mayya-sharipova]

It looks like this test was run against 6.8 mixed cluster:
./gradlew ':qa:mixed-cluster:v6.8.7#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/110_field_collapsing/field collapsing and inner_hits}" -Dtests.seed=BFA11AE1AA03B68A -Dtests.security.manager=true -Dtests.locale=ar-MA -Dtests.timezone=Australia/Sydney -Dcompiler.java=13

This doesn't reproduce for me.

It doesn't reproduce for me just on 7.x cluster either:
./gradlew ':distribution:archives:integ-test-zip:integTestRunner' --tests "org.elasticsearch.test.rest.IntegTestZipClientYamlTestSuiteIT.test {p0=search/110_field_collapsing/field collapsing and inner_hits}"

The stashed output from the failed test is:

  "stash" : {
    "body" : {
      "took" : 12,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 5,
        "max_score" : null,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "6",
            "_score" : null,
            "_source" : {
              "sort" : 36,
              "numeric_group" : 3
            },
            "fields" : {
              "numeric_group" : [
                3
              ]
            },
            "sort" : [
              36
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 1,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "6",
                      "_score" : null,
                      "_source" : {
                        "sort" : 36,
                        "numeric_group" : 3
                      },
                      "sort" : [
                        36
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "3",
            "_score" : null,
            "_source" : {
              "sort" : 24,
              "numeric_group" : 1
            },
            "fields" : {
              "numeric_group" : [
                1
              ]
            },
            "sort" : [
              24
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 3,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "2",
                      "_score" : null,
                      "_source" : {
                        "sort" : 6,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        6
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "1",
                      "_score" : "-Infinity",
                      "_source" : {
                        "sort" : 10,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        10
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "4",
            "_score" : null,
            "_source" : {
              "sort" : 10,
              "numeric_group" : 25
            },
            "fields" : {
              "numeric_group" : [
                25
              ]
            },
            "sort" : [
              10
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 2,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "5",
                      "_score" : "-Infinity",
                      "_source" : {
                        "sort" : 5,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        5
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "4",
                      "_score" : null,
                      "_source" : {
                        "sort" : 10,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        10
                      ]
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  }
}]

As expected, 3 collapsed groups returned with hits.total: 1, 3 and 2 which should sum up to 6, but incorrectly summed up to 5.
I wonder if the failure is related to one of the hits having "_score" : "-Infinity"

[mayya-sharipova]

There were no failures of this test after Feb 17.

There is another failure of this test from Jan 27 but on different condition:
https://gradle-enterprise.elastic.co/s/j3kbaua2fjy6s

java.lang.AssertionError: hits.hits.1.inner_hits.sub_hits.hits.total didn't match expected value: |
hits.hits.1.inner_hits.sub_hits.hits.total: expected Integer [3] but was Integer [2]

Here in the missing hit in the second group for inner hits, a doc with "_id" : "2", "sort" : 6 .

[{
  "stash" : {
    "body" : {
      "took" : 32,
      "timed_out" : false,
      "_shards" : {
        "total" : 5,
        "successful" : 5,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 6,
        "max_score" : null,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "6",
            "_score" : "-Infinity",
            "_source" : {
              "sort" : 36,
              "numeric_group" : 3
            },
            "fields" : {
              "numeric_group" : [
                3
              ]
            },
            "sort" : [
              36
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 1,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "6",
                      "_score" : "-Infinity",
                      "_source" : {
                        "sort" : 36,
                        "numeric_group" : 3
                      },
                      "sort" : [
                        36
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "3",
            "_score" : null,
            "_source" : {
              "sort" : 24,
              "numeric_group" : 1
            },
            "fields" : {
              "numeric_group" : [
                1
              ]
            },
            "sort" : [
              24
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 2,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "2",
                      "_score" : "-Infinity",
                      "_source" : {
                        "sort" : 6,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        6
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "3",
                      "_score" : null,
                      "_source" : {
                        "sort" : 24,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        24
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "4",
            "_score" : "-Infinity",
            "_source" : {
              "sort" : 10,
              "numeric_group" : 25
            },
            "fields" : {
              "numeric_group" : [
                25
              ]
            },
            "sort" : [
              10
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 2,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "5",
                      "_score" : null,
                      "_source" : {
                        "sort" : 5,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        5
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "4",
                      "_score" : "-Infinity",
                      "_source" : {
                        "sort" : 10,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        10
                      ]
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  }
}]

[mayya-sharipova]

Another failure from Nov 1, 2019
https://gradle-enterprise.elastic.co/s/uztw7loist4la

Here there is also a missing document in the second group 1 for inner hits, a doc with "_id" : "3", "sort" : 24.

[{
  "stash" : {
    "body" : {
      "took" : 18,
      "timed_out" : false,
      "_shards" : {
        "total" : 2,
        "successful" : 2,
        "skipped" : 0,
        "failed" : 0
      },
      "hits" : {
        "total" : 5,
        "max_score" : null,
        "hits" : [
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "6",
            "_score" : null,
            "_source" : {
              "sort" : 36,
              "numeric_group" : 3
            },
            "fields" : {
              "numeric_group" : [
                3
              ]
            },
            "sort" : [
              36
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 1,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "6",
                      "_score" : null,
                      "_source" : {
                        "sort" : 36,
                        "numeric_group" : 3
                      },
                      "sort" : [
                        36
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "1",
            "_score" : null,
            "_source" : {
              "sort" : 10,
              "numeric_group" : 1
            },
            "fields" : {
              "numeric_group" : [
                1
              ]
            },
            "sort" : [
              10
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 3,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "2",
                      "_score" : null,
                      "_source" : {
                        "sort" : 6,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        6
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "1",
                      "_score" : null,
                      "_source" : {
                        "sort" : 10,
                        "numeric_group" : 1
                      },
                      "sort" : [
                        10
                      ]
                    }
                  ]
                }
              }
            }
          },
          {
            "_index" : "test",
            "_type" : "_doc",
            "_id" : "4",
            "_score" : null,
            "_source" : {
              "sort" : 10,
              "numeric_group" : 25
            },
            "fields" : {
              "numeric_group" : [
                25
              ]
            },
            "sort" : [
              10
            ],
            "inner_hits" : {
              "sub_hits" : {
                "hits" : {
                  "total" : 2,
                  "max_score" : null,
                  "hits" : [
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "5",
                      "_score" : null,
                      "_source" : {
                        "sort" : 5,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        5
                      ]
                    },
                    {
                      "_index" : "test",
                      "_type" : "_doc",
                      "_id" : "4",
                      "_score" : null,
                      "_source" : {
                        "sort" : 10,
                        "numeric_group" : 25
                      },
                      "sort" : [
                        10
                      ]
                    }
                  ]
                }
              }
            }
          }
        ]
      }
    }
  }
}]

[mayya-sharipova]

it looks like some docs were missed during collection in some cases during the initial query phase, or in other cases during inner_hits phase.

I am out of ideas how this could happen.

[mayya-sharipova]

Another failure on Apr 3 on 7.x windows.
The test failures are mostly on Windows.

gradlew ':qa:mixed-cluster:v6.8.9#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search/110_field_collapsing/field collapsing}" \
  -Dtests.seed=142BCCC577B41C12 \
  -Dtests.security.manager=true \
  -Dtests.locale=th-TH \
  -Dtests.timezone=Pacific/Enderbury \
  -Dcompiler.java=13

Build scan: https://gradle-enterprise.elastic.co/s/zh4vk2ddkfdc2

java.lang.AssertionError: Failure at [search/110_field_collapsing:67]: hits.total didn't match expected value:
hits.total: expected Integer [6] but was Integer [3]

[{
  1>   "stash" : {
  1>     "body" : {
  1>       "took" : 18,
  1>       "timed_out" : false,
  1>       "_shards" : {
  1>         "total" : 5,
  1>         "successful" : 5,
  1>         "skipped" : 0,
  1>         "failed" : 0
  1>       },
  1>       "hits" : {
  1>         "total" : 3,
  1>         "max_score" : null,
  1>         "hits" : [
  1>           {
  1>             "_index" : "test",
  1>             "_type" : "_doc",
  1>             "_id" : "3",
  1>             "_score" : null,
  1>             "_source" : {
  1>               "sort" : 24,
  1>               "numeric_group" : 1
  1>             },
  1>             "fields" : {
  1>               "numeric_group" : [
  1>                 1
  1>               ]
  1>             },
  1>             "sort" : [
  1>               24
  1>             ]
  1>           },
  1>           {
  1>             "_index" : "test",
  1>             "_type" : "_doc",
  1>             "_id" : "5",
  1>             "_score" : "-Infinity",
  1>             "_source" : {
  1>               "sort" : 5,
  1>               "numeric_group" : 25
  1>             },
  1>             "fields" : {
  1>               "numeric_group" : [
  1>                 25
  1>               ]
  1>             },
  1>             "sort" : [
  1>               5
  1>             ]
  1>           }
  1>         ]
  1>       }
  1>     }
  1>   }
  1> }]

[jtibshirani]

Another instance of this failure just popped up: https://gradle-enterprise.elastic.co/s/xfqfl3g3q6th2