Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] SearchableSnapshotsBlobStoreCacheIntegTests testBlobStoreCache failing #77753

Closed
DaveCTurner opened this issue Sep 15, 2021 · 2 comments
Closed
Assignees
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Sep 15, 2021

Build scan:
https://gradle-enterprise.elastic.co/s/2owfbo4u22gl4/tests/:x-pack:plugin:searchable-snapshots:internalClusterTest/org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests/testBlobStoreCache

Reproduction line:
./gradlew ':x-pack:plugin:searchable-snapshots:internalClusterTest' --tests "org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache" -Dtests.seed=2226FF7B586DB45A -Dtests.locale=ar-SY -Dtests.timezone=Australia/Victoria -Druntime.java=11

Applicable branches:
7.14, master

Reproduces locally?:
Yes (same repro line works for master)

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests&tests.test=testBlobStoreCache

Failure excerpt:

java.lang.AssertionError: unexpected

  at __randomizedtesting.SeedInfo.seed([2226FF7B586DB45A:3A6BC2A1CE0F46AD]:0)
  at org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests.refreshSystemIndex(SearchableSnapshotsBlobStoreCacheIntegTests.java:352)
  at org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests.lambda$testBlobStoreCache$2(SearchableSnapshotsBlobStoreCacheIntegTests.java:238)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1035)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1008)
  at org.elasticsearch.xpack.searchablesnapshots.cache.blob.SearchableSnapshotsBlobStoreCacheIntegTests.testBlobStoreCache(SearchableSnapshotsBlobStoreCacheIntegTests.java:237)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:566)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
  at java.lang.Thread.run(Thread.java:829)

  Caused by: org.elasticsearch.index.IndexNotFoundException: no such index [.snapshot-blob-cache]

    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.indexNotFoundException(IndexNameExpressionResolver.java:1023)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.innerResolve(IndexNameExpressionResolver.java:960)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:916)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:219)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:197)
    at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndexNames(IndexNameExpressionResolver.java:83)
    at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.shards(TransportBroadcastReplicationAction.java:115)
    at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:62)
    at org.elasticsearch.action.support.replication.TransportBroadcastReplicationAction.doExecute(TransportBroadcastReplicationAction.java:41)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:53)
    at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:49)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:61)
    at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:201)
    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:100)
    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:82)
    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:710)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:129)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:104)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:69)
    at org.elasticsearch.transport.nio.MockNioTransport$MockTcpReadWriteHandler.consumeReads(MockNioTransport.java:312)
    at org.elasticsearch.nio.SocketChannelContext.handleReadBytes(SocketChannelContext.java:217)
    at org.elasticsearch.nio.BytesChannelContext.read(BytesChannelContext.java:29)
    at org.elasticsearch.nio.EventHandler.handleRead(EventHandler.java:128)
    at org.elasticsearch.transport.nio.TestEventHandler.handleRead(TestEventHandler.java:140)
    at org.elasticsearch.nio.NioSelector.handleRead(NioSelector.java:409)
    at org.elasticsearch.nio.NioSelector.processKey(NioSelector.java:235)
    at org.elasticsearch.nio.NioSelector.singleLoop(NioSelector.java:163)
    at org.elasticsearch.nio.NioSelector.runLoop(NioSelector.java:120)
    at java.lang.Thread.run(Thread.java:829)

@DaveCTurner DaveCTurner added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Sep 15, 2021
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team label Sep 15, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@tlrx tlrx self-assigned this Sep 15, 2021
DaveCTurner added a commit that referenced this issue Sep 15, 2021
DaveCTurner added a commit that referenced this issue Sep 15, 2021
DaveCTurner added a commit that referenced this issue Sep 15, 2021
DaveCTurner added a commit that referenced this issue Sep 15, 2021
@DaveCTurner
Copy link
Contributor Author

Muted in master, 7.x, 7.15 and 7.14.

tlrx added a commit to tlrx/elasticsearch that referenced this issue Sep 15, 2021
This test sometimes fails (elastic#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This pull request adjusts the test to always index at least
some docs.
@tlrx tlrx closed this as completed in 9275eca Sep 16, 2021
tlrx added a commit to tlrx/elasticsearch that referenced this issue Sep 16, 2021
…elastic#77758)

This test sometimes fails (elastic#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of elastic#77758
tlrx added a commit to tlrx/elasticsearch that referenced this issue Sep 16, 2021
…elastic#77758)

This test sometimes fails (elastic#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of elastic#77758
tlrx added a commit to tlrx/elasticsearch that referenced this issue Sep 16, 2021
…elastic#77758)

This test sometimes fails (elastic#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of elastic#77758
elasticsearchmachine pushed a commit that referenced this issue Sep 16, 2021
…#77758) (#77848)

This test sometimes fails (#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of #77758
elasticsearchmachine pushed a commit that referenced this issue Sep 16, 2021
…#77758) (#77843)

This test sometimes fails (#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of #77758
elasticsearchmachine pushed a commit that referenced this issue Sep 16, 2021
…#77758) (#77844)

This test sometimes fails (#77753) when it tries to refresh the
snapshot blob cache index while it wasn't created at all due to
the test index being completely empty with 0 documents indexed.

Having an empty index in this test does not make much sense
because the index will contain no segments and therefore no
segments file to cache in the system index.

This commit adjusts the test to always index at least
some docs.

Backport of #77758

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Development

No branches or pull requests

3 participants