-
Notifications
You must be signed in to change notification settings - Fork 573
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Distributor/Ingestor: Replace out of order validation with ingestion window #2452
Distributor/Ingestor: Replace out of order validation with ingestion window #2452
Conversation
48d4a31
to
f14b9ba
Compare
f14b9ba
to
9bbb6f4
Compare
I'm wondering if we should do anything about the order of the profiles in memory because I think it's necessary for deduplication. I see that profiles are sorted when we flush a row group on disk, but I couldn't find if we sort in-memory profiles explicitly at query time. I expected to find it here, but maybe I don't fully understand how it works. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good
should we remove delta as well? just read the design doc
9bbb6f4
to
fa5f8e9
Compare
fa5f8e9
to
06b45e4
Compare
06b45e4
to
fb00961
Compare
@kolesnikovae you have raised a good point, I have added a binary search for the cases when profiles get added out of order. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Yep we need to keep the in memory sorted.
This change removes of order validation and instead make sure the ingested profiles are within the ingestion window, which will by default reach from now-1h to now+10m.
This will ensure a smoother experience with workloads that don't have unique labels sets