Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add step param to Patterns Query API #12703

Merged
merged 6 commits into from
Apr 25, 2024
Merged

Conversation

benclive
Copy link
Contributor

What this PR does / why we need it:

  • Adds an optional step parameter to the Patterns Query API
  • This returns Pattern counts aggregated by the step duration instead of the raw samples we currently store.
    • Patterns continue to be stored in 10 second samples in the backend as this PR this only affects the read path.

Which issue(s) this PR fixes:
Fixes #12689

Special notes for your reviewer:

  • There may be some aggregation code elsewhere in Loki that can be re-used here.
  • Step is currently unused by the frontend but will default to a minimum of 10 seconds (the sample resolution) if not specified.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

pkg/pattern/drain/chunk.go Show resolved Hide resolved
{Timestamp: 1, Value: 2},
{Timestamp: 3, Value: 4},
{Timestamp: 5, Value: 6},
{Timestamp: 2, Value: 2},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The stepping functionality shunts values into the step bucket before the sample TS, so these tests all had to be skewed down by to an even numbered millisecond as the step size is 2ms. Let me know if you don't agree with this behaviour.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, but you could have made the step configurable in the tests and default to previous behaviour.

@benclive benclive marked this pull request as ready for review April 19, 2024 16:19
@benclive benclive requested a review from a team as a code owner April 19, 2024 16:19
@cyriltovena
Copy link
Contributor

I think there updates requires in those files too.

https://github.com/grafana/loki/blob/main/pkg/loghttp/patterns.go#L9
https://github.com/grafana/loki/blob/main/pkg/querier/queryrange/codec.go#L999

The frontend in distributed mode is a huge minefields, I suggest you give a go and test it in dev to make sure it works correctly. Hit me up if you need help for doing so.

Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

The http parsing could use some tests but nothing blocking

@cyriltovena cyriltovena merged commit 7b8533e into main Apr 25, 2024
58 checks passed
@cyriltovena cyriltovena deleted the add-step-to-patterns-api branch April 25, 2024 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add step into the Pattern API
2 participants