MixedClusterClientYamlTestSuiteIT: Multisearch test with typed_keys parameter for sampler and significant terms failure #57402

davidkyle · 2020-06-01T09:38:38Z

Build scan:
https://gradle-enterprise.elastic.co/s/qaucfxmv5kp76

Repro line:

 ./gradlew ':qa:mixed-cluster:v6.8.10#mixedClusterTest' --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=msearch/20_typed_keys/Multisearch test with typed_keys parameter for sampler and significant terms}" \
  -Dtests.seed=A0B72CC161F91CA5 \
  -Dtests.security.manager=true \
  -Dtests.locale=is \
  -Dtests.timezone=Indian/Mahe \
  -Druntime.java=8

Reproduces locally?:
No

Applicable branches:
7.x

Failure history:
Many failures since May 30th all on the 7.x branch in a mixed cluster with 6.8.y

https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-7d,mode:quick,to:now))&_a=(columns:!(branch),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'%22Multisearch%20test%22'),sort:!(time,desc))

Failure excerpt:

java.lang.AssertionError: Failure at [msearch/20_typed_keys:107]: responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count didn't match expected value:
responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count: expected Integer [5] but was Integer [4]

	at __randomizedtesting.SeedInfo.seed([A0B72CC161F91CA5:28E3131BCF05715D]:0)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:405)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:382)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.AssertionError: responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count didn't match expected value:
responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count: expected Integer [5] but was Integer [4]

	at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:93)
	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:398)
	... 37 more

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-06-01T09:38:40Z

Pinging @elastic/es-search (:Search/Search)

For #57402

davidkyle · 2020-06-01T12:03:21Z

Muted on 7.x in #57406

elasticmachine · 2020-06-03T19:58:23Z

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

jimczi · 2020-06-03T20:00:12Z

@nik9000 could it be related to your recent change on the sig terms agg ?

nik9000 · 2020-06-03T21:28:20Z

I'll take a look!

nik9000 · 2020-06-03T21:38:57Z

Yup. That is me! I've got a fix coming in #57397.

nik9000 · 2020-06-03T21:51:14Z

This reproduces it for me:

while ./gradlew -p qa/mixed-cluster/ clean 'v6.8.10#mixedClusterTest' --tests "*significant terms*"   -Dtests.seed=A0B72CC161F91CA5   -Dtests.security.manager=true   -Dtests.locale=is   -Dtests.timezone=Indian/Mahe   -Druntime.java=8 -i; do echo ok; done

Fixed in 98c379c. Closes #57402

droberts195 · 2020-06-05T13:20:40Z

This is failing again in the 7.x branch even after the fix of 98c379c.

Two checks on the same PR build failed with this error:

java.lang.AssertionError: Failure at [msearch/20_typed_keys:107]: responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count didn't match expected value:
responses.0.aggregations.sampler#test_sampler.sigsterms#test_significant_terms.doc_count: expected Integer [5] but was Integer [4]

nik9000 · 2020-06-05T14:51:31Z

Booo! I see my mistake. Sorry for the noise. I'll push a fix once I'm well and truly sure I'm not doing it wrong.

Tracked in #57402

nik9000 · 2020-06-05T16:04:55Z

Booo! I see my mistake. Sorry for the noise. I'll push a fix once I'm well and truly sure I'm not doing it wrong.

Or, no, I don't see my problem. But I'll figure it out.

nik9000 · 2020-06-05T16:19:52Z

Looks like this happens in master too, but it is less likely to show up:
https://build-stats.elastic.co/app/kibana#/doc/b646ed00-7efc-11e8-bf69-63c8ef516157/build-*/t?id=20200602054604-EAA5CFE2&_g=()

When you run a `significant_terms` aggregation on a field and it *is* mapped but there aren't any values for it then the count of the documents that match the query on that shard still have to be added to the overall doc count. I broke that in elastic#57361. This fixes that. Closes elastic#57402

When you run a `significant_terms` aggregation on a field and it *is* mapped but there aren't any values for it then the count of the documents that match the query on that shard still have to be added to the overall doc count. I broke that in #57361. This fixes that. Closes #57402

When you run a `significant_terms` aggregation on a field and it *is* mapped but there aren't any values for it then the count of the documents that match the query on that shard still have to be added to the overall doc count. I broke that in elastic#57361. This fixes that. Closes elastic#57402

When you run a `significant_terms` aggregation on a field and it *is* mapped but there aren't any values for it then the count of the documents that match the query on that shard still have to be added to the overall doc count. I broke that in #57361. This fixes that. Closes #57402

davidkyle added :Search/Search Search-related issues that do not fall into other categories >test-failure Triaged test failures from CI labels Jun 1, 2020

elasticmachine added the Team:Search Meta label for search team label Jun 1, 2020

davidkyle mentioned this issue Jun 1, 2020

Mute mixed cluster msearch yml test #57406

Merged

davidkyle added a commit that referenced this issue Jun 1, 2020

Mute msearch yml test (#57406)

82f27ef

For #57402

jimczi added the :Analytics/Aggregations Aggregations label Jun 3, 2020

elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 3, 2020

jimczi removed :Search/Search Search-related issues that do not fall into other categories Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Jun 3, 2020

elasticmachine removed the Team:Search Meta label for search team label Jun 3, 2020

jimczi added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Jun 3, 2020

nik9000 self-assigned this Jun 3, 2020

nik9000 added a commit that referenced this issue Jun 4, 2020

Drop skip on test after backporting fix

de27253

Fixed in 98c379c. Closes #57402

nik9000 closed this as completed Jun 4, 2020

droberts195 reopened this Jun 5, 2020

nik9000 added a commit that referenced this issue Jun 5, 2020

Re-mute test

94b3eed

Tracked in #57402

nik9000 mentioned this issue Jun 5, 2020

Fix a bug with missing fields in sig_terms #57757

Merged

nik9000 mentioned this issue Jun 8, 2020

Fix a bug with missing fields in sig_terms (forward port of #57757) #57817

Merged

nik9000 closed this as completed in #57817 Jun 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MixedClusterClientYamlTestSuiteIT: Multisearch test with typed_keys parameter for sampler and significant terms failure #57402

MixedClusterClientYamlTestSuiteIT: Multisearch test with typed_keys parameter for sampler and significant terms failure #57402

davidkyle commented Jun 1, 2020 •

edited by nik9000

elasticmachine commented Jun 1, 2020

davidkyle commented Jun 1, 2020

elasticmachine commented Jun 3, 2020

jimczi commented Jun 3, 2020

nik9000 commented Jun 3, 2020

nik9000 commented Jun 3, 2020

nik9000 commented Jun 3, 2020

droberts195 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

MixedClusterClientYamlTestSuiteIT: Multisearch test with typed_keys parameter for sampler and significant terms failure #57402

MixedClusterClientYamlTestSuiteIT: Multisearch test with typed_keys parameter for sampler and significant terms failure #57402

Comments

davidkyle commented Jun 1, 2020 • edited by nik9000

elasticmachine commented Jun 1, 2020

davidkyle commented Jun 1, 2020

elasticmachine commented Jun 3, 2020

jimczi commented Jun 3, 2020

nik9000 commented Jun 3, 2020

nik9000 commented Jun 3, 2020

nik9000 commented Jun 3, 2020

droberts195 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

nik9000 commented Jun 5, 2020

davidkyle commented Jun 1, 2020 •

edited by nik9000