Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot failing #55513

Closed
iverase opened this issue Apr 21, 2020 · 2 comments · Fixed by #55657
Closed

[CI] SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot failing #55513

iverase opened this issue Apr 21, 2020 · 2 comments · Fixed by #55657
Assignees
@original-brownbear
Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI

Comments

@iverase
Copy link
Contributor

iverase commented Apr 21, 2020

There are a couple of errors for this test in the last week:

https://gradle-enterprise.elastic.co/s/iqg3cqcqalyak
https://gradle-enterprise.elastic.co/s/3ilvp7ty52ybk

Both seems to fail with the same error:

 Caused by:
        java.lang.AssertionError: fake allocation id has to be removed, inSyncAllocationIds:[AWIAuDNXRKK3b0jtoJ-ePw, _forced_allocation_]
            at __randomizedtesting.SeedInfo.seed([FBB4D9C024EA5F0A]:0)
            at org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater.updateInSyncAllocations(IndexMetadataUpdater.java:186)
            at org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater.applyChanges(IndexMetadataUpdater.java:122)
            at org.elasticsearch.cluster.routing.allocation.RoutingAllocation.updateMetadataWithRoutingChanges(RoutingAllocation.java:238)
            at org.elasticsearch.cluster.routing.allocation.AllocationService.buildResult(AllocationService.java:147)
            at org.elasticsearch.cluster.routing.allocation.AllocationService.buildResultAndLogHealthChange(AllocationService.java:132)
            at org.elasticsearch.cluster.routing.allocation.AllocationService.applyStartedShards(AllocationService.java:128)
            at org.elasticsearch.cluster.action.shard.ShardStateAction$ShardStartedClusterStateTaskExecutor.execute(ShardStateAction.java:579)
            at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:697)
            at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:319)
            at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:214)
            at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)
            at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
            at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:632)
            at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
            at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:834)

I have not been able to reproduce it:

./gradlew ':x-pack:plugin:searchable-snapshots:test' --tests "org.elasticsearch.xpack.searchablesnapshots.SearchableSnapshotsIntegTests.testCreateAndRestoreSearchableSnapshot"   -Dtests.seed=38EA7CA2AD868AF3   -Dtests.security.manager=true   -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP   -Dtests.timezone=Navajo   -Dcompiler.java=14
@iverase iverase added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI labels Apr 21, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

@original-brownbear original-brownbear self-assigned this Apr 22, 2020
@original-brownbear
Copy link
Member

This is reproducible locally at a fairly low rate (~1/1k empirically). Still tracking down how we get to this point.

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Apr 23, 2020
@original-brownbear
Fix Broken ExistingStoreRecoverySource Deserialization
1b3879e 
We are using `FORCE_STALE_PRIMARY_INSTANCE` in instance equality checks `==`
but were creating new instances of `ExistingStoreRecoverySource` when reading
from the wire. This could break these checks in corner cases, causing
`org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater#shardStarted`
to not remove the force allocation fake id when starting a shard.

Closes elastic#55513
@original-brownbear original-brownbear mentioned this issue Apr 23, 2020
Merged
original-brownbear added a commit that referenced this issue Apr 23, 2020
@original-brownbear
Fix Broken ExistingStoreRecoverySource Deserialization (#55657)
1804396 
We are using `FORCE_STALE_PRIMARY_INSTANCE` in instance equality checks `==`
but were creating new instances of `ExistingStoreRecoverySource` when reading
from the wire. This could break these checks in corner cases, causing
`org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater#shardStarted`
to not remove the force allocation fake id when starting a shard.

Closes #55513
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Apr 23, 2020
@original-brownbear
Fix Broken ExistingStoreRecoverySource Deserialization (elastic#55657)
1a5496c 
We are using `FORCE_STALE_PRIMARY_INSTANCE` in instance equality checks `==`
but were creating new instances of `ExistingStoreRecoverySource` when reading
from the wire. This could break these checks in corner cases, causing
`org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater#shardStarted`
to not remove the force allocation fake id when starting a shard.

Closes elastic#55513
@original-brownbear original-brownbear mentioned this issue Apr 23, 2020
Merged
original-brownbear added a commit that referenced this issue Apr 23, 2020
@original-brownbear
Fix Broken ExistingStoreRecoverySource Deserialization (#55657) (#55665)
dc89978 
We are using `FORCE_STALE_PRIMARY_INSTANCE` in instance equality checks `==`
but were creating new instances of `ExistingStoreRecoverySource` when reading
from the wire. This could break these checks in corner cases, causing
`org.elasticsearch.cluster.routing.allocation.IndexMetadataUpdater#shardStarted`
to not remove the force allocation fake id when starting a shard.

Closes #55513
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@original-brownbear original-brownbear

Labels
:Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Broken ExistingStoreRecoverySource Deserialization original-brownbear/elasticsearch
3 participants
@original-brownbear @elasticmachine @iverase