Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

storage-aggregation.conf parsing fixes #1989

Merged
merged 8 commits into from
Jun 22, 2021
Merged

storage-aggregation.conf parsing fixes #1989

merged 8 commits into from
Jun 22, 2021

Conversation

Dieterbe
Copy link
Contributor

@Dieterbe Dieterbe commented Jun 17, 2021

  1. use grafana fork of configparser library.
    why fork it? well, to review this properly, you'll have to look at all the commits that https://github.com/grafana/configparser adds on top of upstream, but in particular
  1. add a bunch of unit tests (to test our storage-aggregation.conf use case on top of configparser)

  2. thanks to the fork, fix a bunch of issues:

  • implement defaults like in carbon:
    • xFilesFactor of 0.5 (note that metrictank didn't actually implement xFF so us incorrectly setting this field was not a problem)
    • aggregationMethod avg (note that on grafanacloud and everywhere else that I know of, people always explicitly set this field, but let's be correct and do it like carbon)
  • handle special cases better (e.g. empty file -> use defaults), without needing to do ugly hacks

Why do we need this now?
it's not critical to fix this for metrictank now, however i'ld like to use this library in carbon-relay-ng, and there we must accommodate whatever files the user throw at us.

@fkaleo fkaleo self-requested a review June 18, 2021 08:56
conf/aggregations.go Outdated Show resolved Hide resolved
Copy link
Contributor

@fkaleo fkaleo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything checks out. Thank you!

BEFORE:
    aggregations_test.go:318: testcase "defaults with some comments" expected
        exp agg {Data:[{Name:foo Pattern:foo.* XFilesFactor:0.5 AggregationMethod:[avg]}] DefaultAggregation:{Name:default Pattern:.* XFilesFactor:0.5 AggregationMethod:[avg]}}
        got agg {Data:[{Name:foo Pattern:foo.* # another comment here XFilesFactor:0.5 AggregationMethod:[avg]}] DefaultAggregation:{Name:default Pattern:.* XFilesFactor:0.5 AggregationMethod:[avg]}}
AFTER:
    aggregations_test.go:320: testcase "defaults with some comments" mismatch (-want +got):
          conf.Aggregations(
        - 	{
        - 		Data: []conf.Aggregation{
        - 			{
        - 				Name:              "foo",
        - 				Pattern:           s"foo.*",
        - 				XFilesFactor:      0.5,
        - 				AggregationMethod: []conf.Method{s"avg"},
        - 			},
        - 		},
        - 		DefaultAggregation: conf.Aggregation{
        - 			Name:              "default",
        - 			Pattern:           s".*",
        - 			XFilesFactor:      0.5,
        - 			AggregationMethod: []conf.Method{s"avg"},
        - 		},
        - 	},
        + 	{
        + 		Data: []conf.Aggregation{
        + 			{
        + 				Name:              "foo",
        + 				Pattern:           s"foo.* # another comment here",
        + 				XFilesFactor:      0.5,
        + 				AggregationMethod: []conf.Method{s"avg"},
        + 			},
        + 		},
        + 		DefaultAggregation: conf.Aggregation{
        + 			Name:              "default",
        + 			Pattern:           s".*",
        + 			XFilesFactor:      0.5,
        + 			AggregationMethod: []conf.Method{s"avg"},
        + 		},
        + 	},
          )
@Dieterbe Dieterbe merged commit 971d683 into master Jun 22, 2021
@Dieterbe Dieterbe deleted the storage-aggregation branch June 22, 2021 08:18
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants