Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align series by step in *seriesLists functions if steps or value length not equal #99

Merged
merged 1 commit into from
Sep 7, 2022

Conversation

carrieedwards
Copy link
Collaborator

@carrieedwards carrieedwards commented Sep 7, 2022

This PR addresses a bug with the *seriesLists functions, such as divideSeriesList. This function has been returning an error if the step times of the numerator and denominator are not equal, or if the length of the values in each series are not equal. This is not consistent with Graphite web's implementation, which finds the LCM of the step and then consolidates based on that step time. The numerators and denominators should now be aligned by step, and should be aligned when the number of values are different, and no error should be returned for mismatched step times or value lengths between series.

@carrieedwards carrieedwards linked an issue Sep 7, 2022 that may be closed by this pull request
@grafanabot

This comment has been minimized.

@carrieedwards carrieedwards marked this pull request as ready for review September 7, 2022 17:51
Copy link

@npazosmendez npazosmendez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, just a small comment regarding the step calculation

also nit: can we update the name of the PR so it reflects the fact that this also takes care of different lengths? for future reference

@@ -133,6 +132,7 @@ func (f *seriesList) Do(ctx context.Context, e parser.Expr, from, until int64, v
}

var denominator *types.MetricData
step, alignStep := helper.GetCommonStep(append(numerators, denominators...))

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the step should be calculated for each pair of series and not for all of them

Comment on lines 154 to 158
if alignStep || len(numerator.Values) != len(denominator.Values) {
alignedSeries := helper.ScaleToCommonStep(types.CopyMetricDataSlice([]*types.MetricData{numerator, denominator}), step)
numerator = alignedSeries[0]
denominator = alignedSeries[1]

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following up on my other comment, I think I would calculate the step and alignStep here:

pair := []*types.MetricData{numerator, denominator})
step, alignStep := helper.GetCommonStep(pair)
if alignStep || len(numerator.Values) != len(denominator.Values) {

....

@carrieedwards carrieedwards changed the title Align series by step in divideBySeries if steps not equal Align series by step in *seriesLists functions if steps not equal Sep 7, 2022
@grafanabot

This comment has been minimized.

@grafanabot
Copy link

Go coverage report:

Click to expand.
File %
github.com/go-graphite/carbonapi/cmd/carbonapi/http/capability.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/enrichcontext.go 6.7%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/find_handlers.go 33.5%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/functions_handler.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/helper.go 27.3%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/info_handlers.go 53.8%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/init.go 84.6%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/lbcheck_handler.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/metrics.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/render_handler.go 50.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/tags_handler.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/usage_handler.go 0.0%
github.com/go-graphite/carbonapi/cmd/carbonapi/http/version_handler.go 0.0%
github.com/go-graphite/carbonapi/date/date.go 78.9%
github.com/go-graphite/carbonapi/expr/consolidations/consolidations.go 44.5%
github.com/go-graphite/carbonapi/expr/expr.go 30.0%
github.com/go-graphite/carbonapi/expr/functions/absolute/function.go 80.0%
github.com/go-graphite/carbonapi/expr/functions/aggregate/function.go 82.2%
github.com/go-graphite/carbonapi/expr/functions/aggregateLine/function.go 81.0%
github.com/go-graphite/carbonapi/expr/functions/aggregateSeriesLists/function.go 73.5%
github.com/go-graphite/carbonapi/expr/functions/aggregateWithWildcards/function.go 86.7%
github.com/go-graphite/carbonapi/expr/functions/alias/function.go 83.3%
github.com/go-graphite/carbonapi/expr/functions/aliasByBase64/function.go 88.9%
github.com/go-graphite/carbonapi/expr/functions/aliasByMetric/function.go 92.9%
github.com/go-graphite/carbonapi/expr/functions/aliasByNode/function.go 83.3%
github.com/go-graphite/carbonapi/expr/functions/aliasByRedis/function.go 72.1%
github.com/go-graphite/carbonapi/expr/functions/aliasSub/function.go 77.4%
github.com/go-graphite/carbonapi/expr/functions/asPercent/function.go 77.4%
github.com/go-graphite/carbonapi/expr/functions/averageOutsidePercentile/function.go 89.3%
github.com/go-graphite/carbonapi/expr/functions/averageSeriesWithWildcards/function.go 89.4%
github.com/go-graphite/carbonapi/expr/functions/below/function.go 91.7%
github.com/go-graphite/carbonapi/expr/functions/cactiStyle/function.go 88.7%
github.com/go-graphite/carbonapi/expr/functions/changed/function.go 92.3%
github.com/go-graphite/carbonapi/expr/functions/constantLine/function.go 88.2%
github.com/go-graphite/carbonapi/expr/functions/delay/function.go 90.6%
github.com/go-graphite/carbonapi/expr/functions/derivative/function.go 93.8%
github.com/go-graphite/carbonapi/expr/functions/divideSeries/function.go 88.1%
github.com/go-graphite/carbonapi/expr/functions/ewma/function.go 90.0%
github.com/go-graphite/carbonapi/expr/functions/exclude/function.go 81.8%
github.com/go-graphite/carbonapi/expr/functions/exp/function.go 90.9%
github.com/go-graphite/carbonapi/expr/functions/exponentialMovingAverage/function.go 72.2%
github.com/go-graphite/carbonapi/expr/functions/fallbackSeries/function.go 86.7%
github.com/go-graphite/carbonapi/expr/functions/filter/function.go 84.4%
github.com/go-graphite/carbonapi/expr/functions/grep/function.go 81.8%
github.com/go-graphite/carbonapi/expr/functions/groupByNode/function.go 86.5%
github.com/go-graphite/carbonapi/expr/functions/groupByTags/function.go 90.0%
github.com/go-graphite/carbonapi/expr/functions/heatMap/function.go 86.4%
github.com/go-graphite/carbonapi/expr/functions/heatMap/helpers.go 84.6%
github.com/go-graphite/carbonapi/expr/functions/highestLowest/function.go 87.1%
github.com/go-graphite/carbonapi/expr/functions/hitcount/function.go 93.8%
github.com/go-graphite/carbonapi/expr/functions/identity/function.go 89.5%
github.com/go-graphite/carbonapi/expr/functions/integral/function.go 94.4%
github.com/go-graphite/carbonapi/expr/functions/integralByInterval/function.go 85.4%
github.com/go-graphite/carbonapi/expr/functions/integralWithReset/function.go 86.1%
github.com/go-graphite/carbonapi/expr/functions/interpolate/function.go 92.1%
github.com/go-graphite/carbonapi/expr/functions/invert/function.go 93.3%
github.com/go-graphite/carbonapi/expr/functions/isNotNull/function.go 93.8%
github.com/go-graphite/carbonapi/expr/functions/join/function.go 89.5%
github.com/go-graphite/carbonapi/expr/functions/keepLastValue/function.go 93.2%
github.com/go-graphite/carbonapi/expr/functions/legendValue/function.go 90.9%
github.com/go-graphite/carbonapi/expr/functions/limit/function.go 82.4%
github.com/go-graphite/carbonapi/expr/functions/linearRegression/function.go 80.0%
github.com/go-graphite/carbonapi/expr/functions/logarithm/function.go 90.9%
github.com/go-graphite/carbonapi/expr/functions/logit/function.go 91.3%
github.com/go-graphite/carbonapi/expr/functions/lowPass/function.go 89.3%
github.com/go-graphite/carbonapi/expr/functions/mapSeries/function.go 88.0%
github.com/go-graphite/carbonapi/expr/functions/minMax/function.go 86.7%
github.com/go-graphite/carbonapi/expr/functions/mostDeviant/function.go 88.6%
github.com/go-graphite/carbonapi/expr/functions/moving/function.go 82.1%
github.com/go-graphite/carbonapi/expr/functions/movingMedian/function.go 86.1%
github.com/go-graphite/carbonapi/expr/functions/multiplySeriesWithWildcards/function.go 90.2%
github.com/go-graphite/carbonapi/expr/functions/nPercentile/function.go 90.0%
github.com/go-graphite/carbonapi/expr/functions/nonNegativeDerivative/function.go 90.5%
github.com/go-graphite/carbonapi/expr/functions/offset/function.go 88.0%
github.com/go-graphite/carbonapi/expr/functions/offsetToZero/function.go 94.1%
github.com/go-graphite/carbonapi/expr/functions/pearson/function.go 86.1%
github.com/go-graphite/carbonapi/expr/functions/pearsonClosest/function.go 75.0%
github.com/go-graphite/carbonapi/expr/functions/perSecond/function.go 90.5%
github.com/go-graphite/carbonapi/expr/functions/percentileOfSeries/function.go 78.3%
github.com/go-graphite/carbonapi/expr/functions/polyfit/function.go 87.3%
github.com/go-graphite/carbonapi/expr/functions/pow/function.go 88.9%
github.com/go-graphite/carbonapi/expr/functions/powSeries/function.go 95.0%
github.com/go-graphite/carbonapi/expr/functions/rangeOfSeries/function.go 94.3%
github.com/go-graphite/carbonapi/expr/functions/removeBelowSeries/function.go 92.3%
github.com/go-graphite/carbonapi/expr/functions/removeBetweenPercentile/function.go 89.2%
github.com/go-graphite/carbonapi/expr/functions/removeEmptySeries/function.go 90.0%
github.com/go-graphite/carbonapi/expr/functions/round/function.go 90.0%
github.com/go-graphite/carbonapi/expr/functions/scale/function.go 88.6%
github.com/go-graphite/carbonapi/expr/functions/scaleToSeconds/function.go 88.5%
github.com/go-graphite/carbonapi/expr/functions/seriesList/function.go 67.2%
github.com/go-graphite/carbonapi/expr/functions/sigmoid/function.go 91.3%
github.com/go-graphite/carbonapi/expr/functions/sinFunction/function.go 86.2%
github.com/go-graphite/carbonapi/expr/functions/slo/function.go 84.0%
github.com/go-graphite/carbonapi/expr/functions/slo/helpers.go 72.0%
github.com/go-graphite/carbonapi/expr/functions/smartSummarize/function.go 90.8%
github.com/go-graphite/carbonapi/expr/functions/sortBy/function.go 85.7%
github.com/go-graphite/carbonapi/expr/functions/sortByName/function.go 85.2%
github.com/go-graphite/carbonapi/expr/functions/squareRoot/function.go 90.5%
github.com/go-graphite/carbonapi/expr/functions/substr/function.go 85.4%
github.com/go-graphite/carbonapi/expr/functions/sumSeriesWithWildcards/function.go 89.1%
github.com/go-graphite/carbonapi/expr/functions/summarize/function.go 91.7%
github.com/go-graphite/carbonapi/expr/functions/timeShift/function.go 78.0%
github.com/go-graphite/carbonapi/expr/functions/timeShiftByMetric/function.go 89.7%
github.com/go-graphite/carbonapi/expr/functions/timeShiftByMetric/misc.go 93.8%
github.com/go-graphite/carbonapi/expr/functions/transformNull/function.go 88.5%
github.com/go-graphite/carbonapi/expr/functions/tukey/function.go 80.8%
github.com/go-graphite/carbonapi/expr/functions/unique/function.go 88.9%
github.com/go-graphite/carbonapi/expr/functions/weightedAverage/function.go 85.7%
github.com/go-graphite/carbonapi/expr/helper/align.go 60.7%
github.com/go-graphite/carbonapi/expr/helper/helper.go 9.2%
github.com/go-graphite/carbonapi/expr/helper/metric/extract.go 72.7%
github.com/go-graphite/carbonapi/expr/helper/sort.go 0.0%
github.com/go-graphite/carbonapi/expr/rewrite/aboveSeries/function.go 79.3%
github.com/go-graphite/carbonapi/expr/sort.go 96.1%
github.com/go-graphite/carbonapi/expr/tags/helper.go 93.6%
github.com/go-graphite/carbonapi/expr/types/extract.go 84.6%
github.com/go-graphite/carbonapi/expr/types/list.go 0.0%
github.com/go-graphite/carbonapi/expr/types/metricheap.go 0.0%
github.com/go-graphite/carbonapi/expr/types/types.go 31.5%
github.com/go-graphite/carbonapi/expr/types/windowed.go 0.0%
github.com/go-graphite/carbonapi/pkg/parser/define.go 76.9%
github.com/go-graphite/carbonapi/pkg/parser/interface.go 0.0%
github.com/go-graphite/carbonapi/pkg/parser/internal.go 9.4%
github.com/go-graphite/carbonapi/pkg/parser/interval.go 83.8%
github.com/go-graphite/carbonapi/pkg/parser/parser.go 44.3%
github.com/go-graphite/carbonapi/util/pidfile/pidfile.go 100.0%
github.com/go-graphite/carbonapi/zipper/broadcast/broadcast_group.go 38.5%
github.com/go-graphite/carbonapi/zipper/helper/requests.go 13.3%
github.com/go-graphite/carbonapi/zipper/protocols/graphite/msgpack/type_gen.go 57.2%
github.com/go-graphite/carbonapi/zipper/protocols/irondb/irondb_group.go 2.5%
github.com/go-graphite/carbonapi/zipper/protocols/irondb/irondb_helpers.go 81.1%
github.com/go-graphite/carbonapi/zipper/protocols/prometheus/helpers/helpers.go 17.1%
github.com/go-graphite/carbonapi/zipper/protocols/victoriametrics/feature_set.go 19.7%
github.com/go-graphite/carbonapi/zipper/protocols/victoriametrics/fetch.go 0.0%
github.com/go-graphite/carbonapi/zipper/protocols/victoriametrics/find.go 0.0%
github.com/go-graphite/carbonapi/zipper/protocols/victoriametrics/tags.go 0.0%
github.com/go-graphite/carbonapi/zipper/protocols/victoriametrics/victoriametrics_group.go 7.8%
github.com/go-graphite/carbonapi/zipper/types/backend.go 0.0%
github.com/go-graphite/carbonapi/zipper/types/errors.go 0.0%
github.com/go-graphite/carbonapi/zipper/types/lbmethod.go 0.0%
github.com/go-graphite/carbonapi/zipper/types/requests.go 0.0%
github.com/go-graphite/carbonapi/zipper/types/response.go 6.4%
github.com/go-graphite/carbonapi/zipper/types/stats.go 0.0%
github.com/go-graphite/carbonapi/zipper/zipper.go 0.0%
github.com/go-graphite/carbonapi/zipper/zipper_pb3.go 0.0%
total 56.0%

Go lint report:

No issues found. 😎

Copy link

@npazosmendez npazosmendez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@carrieedwards carrieedwards changed the title Align series by step in *seriesLists functions if steps not equal Align series by step in *seriesLists functions if steps or value length not equal Sep 7, 2022
@carrieedwards carrieedwards merged commit 7b9b44b into main Sep 7, 2022
carrieedwards added a commit that referenced this pull request Sep 14, 2022
…th not equal (#99)

Align series by step in divideBySeries if steps not equal
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DivideSeriesLists not aligning numerators and denominators
3 participants