Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable sharding for avg_over_time. #10373

Merged
merged 7 commits into from Aug 29, 2023

Conversation

jeschkies
Copy link
Contributor

What this PR does / why we need it:
The range aggregation avg_over_time is simply shardable if there's no label reduction.

If there is one it can be express as

sum by (method) (
    sum_over_time(
        {container="app"} | json | unwrap bytes [$__interval])
) 
/
sum by (method) (
    count_over_time(
        {container="app"} | json [$__interval]
    )
) 

Note that sum by, sum_over_time and count_over_time are shardable once more.

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR

Copy link
Member

@owen-d owen-d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Left a nit, but lgtm

@@ -894,6 +903,19 @@ func (r *LogRange) Walk(f WalkFn) {
r.Left.Walk(f)
}

// WithoutUnwrap returns a copy of the log range without the unwrap statement.
func (r *LogRange) WithoutUnwrap() (*LogRange, error) {
left, err := Clone(r.Left)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will be more extensible if we Clone(self) then set the unwrap to nil. This ensures that any future fields are propagated through this function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately it returns and unexpected unwrap error. I'll take a look tomorrow.

pkg/logql/shardmapper.go Outdated Show resolved Hide resolved
Co-authored-by: Owen Diehl <ow.diehl@gmail.com>
Copy link
Contributor

@cstyan cstyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome!

Comment on lines 334 to 335
downstream<sum(avg_over_time({job=~"myapps.*"}|="stats"|jsonbusy="utilization"|unwrapbusy[5m])),shard=0_of_2>
++
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this correct? unwrapbusy missing a space?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and jsonbusy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We remove all white spaces in the tests. I've copied the expected string and adapted. Will fix.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Comment on lines +396 to +399
countOverTimeSelector, err := expr.Left.WithoutUnwrap()
if err != nil {
return nil, 0, err
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for something like avg(avg_over_time({job=~"myapps.*"} |= "stats" | json busy="utilization" | unwrap busy [5m])) is the unwrap part of the left? And then the inner avg_over_time is the right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean in parsing? It has its own field 🙈

@jeschkies jeschkies merged commit f8658fd into grafana:main Aug 29, 2023
4 checks passed
@jeschkies jeschkies deleted the karsten/shard-range-queries branch August 29, 2023 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants