[CI] SnapshotStressTestsIT testRandomActivities failing #104939
Labels
:Distributed/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
medium-risk
An open issue or test failure that is a medium risk to future releases
Team:Distributed
Meta label for distributed team
>test-failure
Triaged test failures from CI
Comments
DaveCTurner
added
:Distributed/Snapshot/Restore
|
Pinging @elastic/es-distributed (Team:Distributed) |
|
I've seen this failure in another test in which we had more verbose logging: fail-2023-12-25T15:49:30.191Z-4dac44f91ae404e81db9861c43e9803243e15c25.log.gz In that case the transition which moved the stuck snapshot to a completed state was an |
DaveCTurner
added a commit
to DaveCTurner/elasticsearch
that referenced
this issue
Feb 7, 2024
Today when processing a batch of `ShardSnapshotUpdate` tasks each update's listener considers whether the corresponding snapshot has completed and, if so, it enqueues it for finalization. This is somewhat inefficient since we may be processing many shard snapshot updates for the same few snapshots, but there's no need to check each snapshot for completion more than once. It's also insufficient since the completion of a shard snapshot may cause the completion of subsequent snapshots too (e.g. they can go from state `QUEUED` straight to `MISSING`). This commit detaches the completion handling from the individual shard snapshot updates and instead makes sure that any snapshot that reaches a completed state is enqueued for finalization. Closes elastic#104939
elasticsearchmachine
pushed a commit
that referenced
this issue
Feb 9, 2024
Today when processing a batch of `ShardSnapshotUpdate` tasks each update's listener considers whether the corresponding snapshot has completed and, if so, it enqueues it for finalization. This is somewhat inefficient since we may be processing many shard snapshot updates for the same few snapshots, but there's no need to check each snapshot for completion more than once. It's also insufficient since the completion of a shard snapshot may cause the completion of subsequent snapshots too (e.g. they can go from state `QUEUED` straight to `MISSING`). This commit detaches the completion handling from the individual shard snapshot updates and instead makes sure that any snapshot that reaches a completed state is enqueued for finalization. Closes #104939
DaveCTurner
added a commit
that referenced
this issue
Feb 9, 2024
Today when processing a batch of `ShardSnapshotUpdate` tasks each update's listener considers whether the corresponding snapshot has completed and, if so, it enqueues it for finalization. This is somewhat inefficient since we may be processing many shard snapshot updates for the same few snapshots, but there's no need to check each snapshot for completion more than once. It's also insufficient since the completion of a shard snapshot may cause the completion of subsequent snapshots too (e.g. they can go from state `QUEUED` straight to `MISSING`). This commit detaches the completion handling from the individual shard snapshot updates and instead makes sure that any snapshot that reaches a completed state is enqueued for finalization. Closes #104939
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Not the same as #104072. The cluster state at failure suggests we've dropped a finalization task somehow (maybe related to a
node-leftevent):Build scan:
https://gradle-enterprise.elastic.co/s/ed4zqzurtbpn6/tests/:server:internalClusterTest/org.elasticsearch.snapshots.SnapshotStressTestsIT/testRandomActivities
Reproduction line:
Applicable branches:
main, 8.12
Reproduces locally?:
Didn't try
Failure history:
Failure dashboard for
org.elasticsearch.snapshots.SnapshotStressTestsIT#testRandomActivitiesFailure excerpt:
The text was updated successfully, but these errors were encountered: