Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

Fix mt-backfill data flush #2043

Merged
merged 4 commits into from
Dec 26, 2022
Merged

Conversation

shanson7
Copy link
Collaborator

@shanson7 shanson7 commented Nov 8, 2022

We have noticed for a while that the mt-backfill tool never seems to flush all of the data it has. It turns out, this is due to a mix of using system time and chunk spans.

AggMetric.lastWrite uses the system time when the data point was ingested, not the timestamp of the datapoint. This is probably the right thing to do to avoid prematurely flushing when backfilling.

However, Aggregator won't GC a chunk until lastWriteTime+agg.span <= chunkMinTs. For rollups with longer spans (say 4h) that means the backfill tool would need to run for 4 hours before it would flush the tail datapoints.

This change is to add a simple ForceGC function so the backfill tool can do flush everything on shutdown. I figured this was the safest way to fix this without changing the behavior of the core components.

@robert-milan robert-milan merged commit 304f5e6 into grafana:master Dec 26, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants