Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement data retention in storage layer #168

Closed
2 of 6 tasks
petethepig opened this issue Apr 28, 2021 · 0 comments · Fixed by #182
Closed
2 of 6 tasks

Implement data retention in storage layer #168

petethepig opened this issue Apr 28, 2021 · 0 comments · Fixed by #182
Labels
backend Mostly go code storage Low level storage matters

Comments

@petethepig
Copy link
Member

petethepig commented Apr 28, 2021

Feature Description

Sometimes it doesn't make sense to store data indefinitely so we need to implement a way to do data retention.

Internally Pyroscope uses segment trees to store the data.
See our Storage Design Doc for more context.

Page2 3

We need to add a new retention threshold parameter that would control how much data (in days) can be stored in pyroscope. Then we would run some sort of a Cleanup method every once in a while to delete old data:

Page3

TODO

  • Implement a method that would delete data before a certain time on segment tree and return a list of trees that need also need to be deleted.
  • cover it with specs
  • add a retention option to config
  • in storage package, put conditions in Put method that would prevent data ingestions past the retention threshold
  • in storage package add a method that would delete segment tree nodes as well as relevant trees
  • create a separate goroutine that would call this cleanup method every X minutes
@petethepig petethepig added backend Mostly go code storage Low level storage matters labels Apr 28, 2021
@petethepig petethepig added this to Q2 (Apr – Jun) 2021 in Pyroscope Public Roadmap Apr 28, 2021
@Rperry2174 Rperry2174 moved this from TODO to Discussion / Planning in Pyroscope Public Roadmap Apr 30, 2021
@Rperry2174 Rperry2174 moved this from Discussion / Seeking Contributors to In progress in Pyroscope Public Roadmap May 15, 2021
@Rperry2174 Rperry2174 moved this from In progress to In Review in Pyroscope Public Roadmap May 21, 2021
@Rperry2174 Rperry2174 moved this from Included in Next Release to Done in Pyroscope Public Roadmap Jul 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend Mostly go code storage Low level storage matters
Development

Successfully merging a pull request may close this issue.

1 participant