Compactor: Adding minimum retention flag validation for downsampling retention #5059
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he mapping between the downsample ranges and their respective resolutions Signed-off-by: mzardab <mzardab@redhat.com>
...if the following conditions are met: * Downsampling is enabled, raw retention must be > 10 days * Downsampling is enabled, 5m resolution retention must be at least 40h * Downsampling is enabled, 1h resolution must be at least 10d This is to avoid unexpected deletions of downsampled/raw data as compactor will only downsample raw blocks after there is enough data to form 2 whole blocks, it will not downsample immediately. Signed-off-by: mzardab <mzardab@redhat.com>
Signed-off-by: mzardab <mzardab@redhat.com>
moadz
commented
Jan 11, 2022
|
|
||
| * Creating 5m downsampling for blocks larger than **40 hours** (2d, 2w) | ||
| * Creating 1h downsampling for blocks larger than **10 days** (2w) | ||
| Compactor is also responsible for downsampling of data. There is a time delay before downsampling at a given resolution is possible. This is necessary because downsampled chunks will have fewer samples in them, and as chunks are fixed size, data spanning more time will be required to fill them. |
There was a problem hiding this comment.
I'm inferring this from the code, 2hr @ 15s sampling to 20hrs @ 5min sampling (~factor of 10). It mentions that we want to produce at least 2 chunks for each downsampling pass, unsure of any other reason for why this would be the case.
yeya24
reviewed
Jan 15, 2022
matej-g
reviewed
Jan 19, 2022
@matej-g just trying to fix some flakey docs tests by rebasing on master. All current feedback has been addressed pending re-review by @yeya24 |
Signed-off-by: mzardab <mzardab@redhat.com>
Signed-off-by: mzardab <mzardab@redhat.com>
Signed-off-by: mzardab <mzardab@redhat.com>
moadz
force-pushed
the
thanos-compactor-retention-resolution-flag-validation
branch
from
a174bb1
to
26d20d9
Compare
bwplotka
approved these changes
Jan 21, 2022
|
|
||
| * Creating 5m downsampling for blocks larger than **40 hours** (2d, 2w) | ||
| * Creating 1h downsampling for blocks larger than **10 days** (2w) | ||
| Compactor is also responsible for downsampling of data. There is a time delay before downsampling at a given resolution is possible. This is necessary because downsampled chunks will have fewer samples in them, and as chunks are fixed size, data spanning more time will be required to fill them. |
Nicholaswang
pushed a commit
to Nicholaswang/thanos
that referenced
this pull request
Mar 6, 2022
…retention (thanos-io#5059) * Renaming `DownsampleRange` -> `ResLevel1DownsampleRange` to specify the mapping between the downsample ranges and their respective resolutions Signed-off-by: mzardab <mzardab@redhat.com> * Adding compactor downsample retention flag validation... ...if the following conditions are met: * Downsampling is enabled, raw retention must be > 10 days * Downsampling is enabled, 5m resolution retention must be at least 40h * Downsampling is enabled, 1h resolution must be at least 10d This is to avoid unexpected deletions of downsampled/raw data as compactor will only downsample raw blocks after there is enough data to form 2 whole blocks, it will not downsample immediately. Signed-off-by: mzardab <mzardab@redhat.com> * Added more clarifying docs on impact of retention on downsampling Signed-off-by: mzardab <mzardab@redhat.com> * Resolving conflicts Signed-off-by: mzardab <mzardab@redhat.com> * Updated CHANGELOG.md Signed-off-by: mzardab <mzardab@redhat.com> * Addressing PR comments Signed-off-by: mzardab <mzardab@redhat.com> Signed-off-by: Nicholaswang <wzhever@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
--retention.resolution-raw> 0--retention.resolution-5m> 0--retention.resolution-1h> 0Verification
Changes based on: #813
tl;dr: For downsampling to happen, blocks must be pass a certain age for the preceding resolution. This was always the case in Compactor but not transparently to users. As such, it's very easy to set up retention in such a way that disrupts downsampling or leads to accidental data loss.