Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compact RAM calculation #722

Closed
arnitolog opened this issue Jan 10, 2019 · 1 comment · Fixed by #1947
Closed

Compact RAM calculation #722

arnitolog opened this issue Jan 10, 2019 · 1 comment · Fixed by #1947

Comments

@arnitolog
Copy link
Contributor

I have one of the blocks which is ~7GB of size. And Compact can't do anything with it. It is OOM killed every time. Is there any way to instruct Compact to process big blocks piecemeal
Here is part of meta.json file:
"version": 1,
"ulid": "01CRYM9RYN78DFKZ73XB4H4WCP",
"minTime": 1537401600000,
"maxTime": 1538611200000,
"stats": {
"numSamples": 4489155614,
"numSeries": 2088751,
"numChunks": 39194144
},
here is graphs for compact pod:
image
here is an error:
level=info ts=2019-01-10T20:46:34.747142839Z caller=compact.go:822 msg="start of GC"
level=info ts=2019-01-10T20:46:34.78464068Z caller=compact.go:207 msg="compaction iterations done"
level=info ts=2019-01-10T20:46:34.784681757Z caller=compact.go:214 msg="start first pass of downsampling"
level=info ts=2019-01-10T20:48:27.547013191Z caller=downsample.go:212 msg="downloaded block" id=01CRYM9RYN78DFKZ73XB4H4WCP duration=1m52.526675454s
fatal error: runtime: out of memory

@GiedriusS
Copy link
Member

GiedriusS commented Jan 4, 2020

Compactor now supports sharding so you could use that as a temporary workaround to reduce RAM usage but we could definitely provide some guidance on how much RAM is needed since it seems like this is a recurring topic: #1928

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants