Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

only fetches one chunk per series in /series #1914

Merged
merged 4 commits into from
Apr 8, 2020

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Apr 7, 2020

What

Improves /series API by fetching a single chunk for each series and then appying matchers one time per series.

Why

Previously we used used the same functions and associated EntryIterator ifc as the traditional query path, fetching all logs in a time range then calculating labels per entry, only deduping them at the end. This was wildly inefficient and would oom queriers.

Proof

Tested locally w/ v9 schema

$ ls -lh /var/log/*.log | grep -v ' 0B ' | wc -l
       9
$ logcli series --match='{filename=~".*log$"}' | wc -l
http://localhost:3100/loki/api/v1/series?end=1586288927776455000&match=%7Bfilename%3D~%22.%2Alog%24%22%7D&start=1586285327776455000
       9

/cc @cyriltovena

pkg/storage/store.go Outdated Show resolved Hide resolved
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, way to go !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants