Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Overwriting points on large series can cause memory spikes during compactions #6557

Closed
jwilder opened this issue May 4, 2016 · 1 comment · Fixed by #6567
Closed

Overwriting points on large series can cause memory spikes during compactions #6557

jwilder opened this issue May 4, 2016 · 1 comment · Fixed by #6567
Labels
area/performance area/tsm
Milestone
1.0.0

Comments

@jwilder
Copy link
Contributor

jwilder commented May 4, 2016

While investigating a query case that could OOM the server (see #6556), it was discovered that compactions likely have the same problem.

If there are a lot of points on disk for the same series spread across many blocks in TSM files and a point is overwritten near the beginning of the shard's time range, the full series could be loaded into RAM triggering OOMs and huge allocations.

The compaction code uses a similar method as the query path to handle updating points in the past in that it just reads the whole series and deduplicates in memory. It's very likely that this can cause the server to OOM during compactions if enough data is being compacted. Compaction code needs to be updated similar to the fix in #6556.
@jwilder jwilder added this to the 1.0.0 milestone May 4, 2016
@andybrier
Copy link

andybrier commented May 5, 2016 •
edited
Loading

I have the same issue

@jwilder jwilder mentioned this issue May 6, 2016
Merged
3 tasks
@jwilder jwilder mentioned this issue May 16, 2016
Merged
2 tasks
@francisdb francisdb mentioned this issue May 17, 2016
Closed
@jwilder jwilder reopened this May 17, 2016
jwilder added a commit that referenced this issue May 17, 2016
@jwilder
Fix memory spike when compacting overwritten points
a574e2e 
If a large series contains a point that is overwritten, the compactor
would load the whole series into RAM during a full compaction.  If
the series was large, it could cause very large RAM spikes and OOMs.

The change reworks the compactor to merge blocks more incrementally
similar to the fix done in #6556.

Fixes #6557
jwilder added a commit that referenced this issue May 18, 2016
@jwilder
Fix memory spike when compacting overwritten points
97e99b3 
If a large series contains a point that is overwritten, the compactor
would load the whole series into RAM during a full compaction.  If
the series was large, it could cause very large RAM spikes and OOMs.

The change reworks the compactor to merge blocks more incrementally
similar to the fix done in #6556.

Fixes #6557
@timhallinflux timhallinflux modified the milestones: 1.0.0, 1.0.0 beta Dec 20, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/performance area/tsm
Projects
None yet
Milestone
1.0.0
Development

Successfully merging a pull request may close this issue.

Fix memory spike when compacting overwritten points influxdata/influxdb
3 participants
@jwilder @andybrier @timhallinflux