Skip to content
This repository has been archived by the owner on Jul 19, 2023. It is now read-only.

Queries and storage maintenance tasks sync #687

Closed
kolesnikovae opened this issue May 12, 2023 · 0 comments · Fixed by #699
Closed

Queries and storage maintenance tasks sync #687

kolesnikovae opened this issue May 12, 2023 · 0 comments · Fixed by #699
Assignees
@kolesnikovae
Labels
area/database kind/performance

Comments

@kolesnikovae
Copy link
Contributor

kolesnikovae commented May 12, 2023 •
edited

When the head block is being flushed on disk it is inaccessible by queries during the operation, and ongoing queries may be blocked for quite some time due to the fact that at some point a write lock is acquired: this causes massive query latencies and timeouts.

The issue is easy to reproduce in a big cluster consisting of dozens and hundreds of ingesters.

Sample trace – reading symbols from memory:
image
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kolesnikovae kolesnikovae

Labels
area/database kind/performance
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix queries and storage block operations synchronisation grafana/phlare
1 participant
@kolesnikovae