Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] DocsClientYamlTestSuiteIT tests failing with "Expected a map containing acknowledged" #87591

Closed
davidkyle opened this issue Jun 13, 2022 · 14 comments · Fixed by #91829
Closed

[CI] DocsClientYamlTestSuiteIT tests failing with "Expected a map containing acknowledged" #87591

davidkyle opened this issue Jun 13, 2022 · 14 comments · Fixed by #91829
Labels
:ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@davidkyle
Copy link
Member

Build scan:
https://gradle-enterprise.elastic.co/s/wz6n46lrlw5g2/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Fml%2Fcommon%2Fapis%2Fget-ml-info%2Fline_38%7D

Reproduction line:
./gradlew ':docs:yamlRestTest' --tests "org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT.test {yaml=reference/ml/common/apis/get-ml-info/line_38}" -Dtests.seed=705165C627C9F35A -Dtests.locale=ro -Dtests.timezone=America/Goose_Bay -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Dreference/ml/common/apis/get-ml-info/line_38%7D

Failure excerpt:

java.lang.AssertionError: Failure at [reference/ml/common/apis/get-ml-info:92]: Expected a map containing
acknowledged: expected <true> but was <missing>
    defaults: <unexpected> but was <{anomaly_detectors={model_memory_limit=1gb, categorization_examples_limit=4, model_snapshot_retention_days=10, daily_model_snapshot_retention_after_days=1, categorization_analyzer={char_filter=[first_line_with_letters], tokenizer=ml_standard, filter=[{type=stop, stopwords=[Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday, Mon, Tue, Wed, Thu, Fri, Sat, Sun, January, February, March, April, May, June, July, August, September, October, November, December, Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec, GMT, UTC]}]}}, datafeeds={scroll_size=1000}}>
upgrade_mode: <unexpected> but was <false>
 native_code: <unexpected> but was <{version=8.4.0-SNAPSHOT, build_hash=2b5d23e8aed603}>
      limits: <unexpected> but was <{total_ml_memory=28968mb, effective_max_model_memory_limit=28928mb}>

  at __randomizedtesting.SeedInfo.seed([705165C627C9F35A:F8055A1C89359EA2]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:510)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:479)
  at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(null:-1)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

  Caused by: java.lang.AssertionError: Expected a map containing
  acknowledged: expected <true> but was <missing>
      defaults: <unexpected> but was <{anomaly_detectors={model_memory_limit=1gb, categorization_examples_limit=4, model_snapshot_retention_days=10, daily_model_snapshot_retention_after_days=1, categorization_analyzer={char_filter=[first_line_with_letters], tokenizer=ml_standard, filter=[{type=stop, stopwords=[Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday, Mon, Tue, Wed, Thu, Fri, Sat, Sun, January, February, March, April, May, June, July, August, September, October, November, December, Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec, GMT, UTC]}]}}, datafeeds={scroll_size=1000}}>
  upgrade_mode: <unexpected> but was <false>
   native_code: <unexpected> but was <{version=8.4.0-SNAPSHOT, build_hash=2b5d23e8aed603}>
        limits: <unexpected> but was <{total_ml_memory=28968mb, effective_max_model_memory_limit=28928mb}>

    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:82)
    at org.elasticsearch.test.MapMatcher.assertMap(MapMatcher.java:65)
    at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:94)
    at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:65)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:499)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:479)
    at jdk.internal.reflect.GeneratedMethodAccessor14.invoke(null:-1)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:568)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
    at java.lang.Thread.run(Thread.java:833)
@davidkyle davidkyle added :ml Machine learning >test-failure Triaged test failures from CI labels Jun 13, 2022
@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label Jun 13, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@droberts195
Copy link
Contributor

It's exactly the same problem as #80409 and #80622 - test fails because it expected an AcknowledgedResponse, but the test code clearly expects a different response and gets that different response that it expected.

There might be some clues in #80409 (comment) and #80409 (comment).

Weirdly the problem went away without any fix being made, but has now come back on Ubuntu 22.04 only.

@pugnascotia
Copy link
Contributor

Another failure: https://gradle-enterprise.elastic.co/s/zpknjtc2hg4yc

@bpintea
Copy link
Contributor

bpintea commented Jun 17, 2022

Another Ubuntu 22.04 failure: https://gradle-enterprise.elastic.co/s/wdswowev3o53g

@davidkyle
Copy link
Member Author

Another failure, Ubuntu again.

https://gradle-enterprise.elastic.co/s/7efnxwgnl5g44/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Fml%2Fcommon%2Fapis%2Fget-ml-memory%2Fline_228%7D

@droberts195
Copy link
Contributor

This hasn't failed recently for the weird reason of the response being as expected but the test comparing it to an acknowledged response. I'll close this for now. Please reopen if the same type of failure occurs again.

@davidkyle davidkyle changed the title [CI] DocsClientYamlTestSuiteIT test {yaml=reference/ml/common/apis/get-ml-info/line_38} failing [CI] DocsClientYamlTestSuiteIT tests failing with "Expected a map containing acknowledged" Oct 20, 2022
@davidkyle
Copy link
Member Author

Reopening with another failure in Ubuntu 20.04.

The actual test is different but the error is the same: the test expected an acknowledged response but something else was returned.

https://gradle-enterprise.elastic.co/s/7bcru4xqthj2i/console-log?task=:docs:yamlRestTest

The issues listed below are effectively duplicates of this one, I'm linking them here for reference:
#80622
#80438
#80409

@davidkyle davidkyle reopened this Oct 20, 2022
@droberts195
Copy link
Contributor

the test expected an acknowledged response but something else was returned

To clarify, the test runner decided it expected an acknowledged response when the expected result configured in the test code was something different and the actual response was exactly what the test code wanted.

This seems to be some weird intermittent problem in the test framework, not in the expected or actual results of any particular test.

This was referenced Nov 16, 2022
Closed
Closed
@ywangd
Copy link
Member

ywangd commented Nov 17, 2022

Another one on 8.5 https://gradle-enterprise.elastic.co/s/p2jzx4vlq2gm2/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Fcluster%2Fnodes-info%2Fline_283%7D

@luigidellaquila
Copy link
Contributor

More failures today: https://gradle-enterprise.elastic.co/s/cclzdvo53h56u/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Fcluster%2Fnodes-info%2Fline_283%7D?top-execution=1

@iverase
Copy link
Contributor

iverase commented Dec 2, 2022

One more in main: https://gradle-enterprise.elastic.co/s/bi4wo2neuurrm

@droberts195 droberts195 mentioned this issue Dec 6, 2022
Closed
@droberts195
Copy link
Contributor

Might be fixed by #91829

@droberts195 droberts195 mentioned this issue Dec 6, 2022
Merged
@thecoop
Copy link
Member

thecoop commented Dec 14, 2022 •
edited

Also seen on 8.5 - https://gradle-enterprise.elastic.co/s/ulfwzad67gk72/tests/:docs:yamlRestTest/org.elasticsearch.smoketest.DocsClientYamlTestSuiteIT/test%20%7Byaml=reference%2Ftext-structure%2Fapis%2Ffind-structure%2Fline_259%7D?top-execution=1 (#91829 was only applied to 8.6 and 8.7)

@thecoop thecoop reopened this Dec 14, 2022
@thecoop thecoop mentioned this issue Dec 14, 2022
Merged
thecoop added a commit that referenced this issue Dec 14, 2022
@thecoop
Mute doc test in find-structure (#92363)
0e8bf7f 
#87591 has not been applied to 8.5
This was referenced Dec 20, 2022
Open
Closed
@droberts195
Copy link
Contributor

8.5 is no longer maintained - closing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix sneaky docs test failure nik9000/elasticsearch
9 participants
@luigidellaquila @bpintea @ywangd @davidkyle @thecoop @droberts195 @pugnascotia @elasticmachine @iverase