Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add acceptance tests #40

Merged
merged 10 commits into from
Apr 20, 2022
Merged

Add acceptance tests #40

merged 10 commits into from
Apr 20, 2022

Conversation

carrieedwards
Copy link
Contributor

This PR adds acceptance tests for influx2cortex

@carrieedwards carrieedwards force-pushed the cedwards/add-acceptance-tests branch 2 times, most recently from 6aaccf7 to 0acacfe Compare April 12, 2022 15:50
Copy link
Contributor

@fionaliao fionaliao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests are looking good 👍

As well as the tests for single points, let's have a test that ingests a list of points (and use the Prometheus query_range endpoint to check all the points have been ingested), and also one where there are multiple series for the same metric but different labels.

acceptance/suite_test.go Show resolved Hide resolved
acceptance/suite_test.go Outdated Show resolved Hide resolved
acceptance/suite_test.go Outdated Show resolved Hide resolved
acceptance/suite_test.go Outdated Show resolved Hide resolved
operations/jsonnet/lib/cortex/cortex.yaml Outdated Show resolved Hide resolved
scripts/compile-commands.sh Outdated Show resolved Hide resolved
scripts/compile-commands.sh Outdated Show resolved Hide resolved
acceptance/write_proxy_test.go Outdated Show resolved Hide resolved
@carrieedwards carrieedwards marked this pull request as ready for review April 14, 2022 14:59
@carrieedwards
Copy link
Contributor Author

On another note, I am going to file a separate PR for changing the write endpoint, since some unit tests will also need to be updated.

Copy link
Contributor

@ywwg ywwg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fantastic, and very clean and easy to understand! thank you. I just would like some more documentation strings so it's a little more approachable, and I have a few other small nits.

acceptance/suite_test.go Show resolved Hide resolved
s.T().Logf("Test ended at %s", time.Now().Format(time.RFC3339Nano))
}

func (s *Suite) printLogs(res *dockertest.Resource) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eventually (not now), is this something that could be broken out into an acceptance test library? This seems to have required a lot of code, and it would be nice if we could reduce the amount of boilerplate required for writing these tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think that would be a good thing to do in the future to limit code duplication

return container
}

func (s *Suite) startInfluxProxy() *dockertest.Resource {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

continuing from above, would it be possible to move the influx-specific setup into a separate file so that we have a clean division between general acceptance test setup and this-proxy-specific stuff?

Value: 23.5,
}

result, _, err := s.api.querierClient.Query(context.Background(), "stat_avg", time.Now())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add a query before the record is written to confirm that the result before and after the write is different?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in, query prometheus to verify that metrics do not exist prior to the write?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct

Value: 3,
}

result, _, err := s.api.querierClient.Query(context.Background(), "{__name__=~\"measurement_.+\",__proxy_source__=\"influx\"}", time.Now())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for all of these I think


registerer.RegisterRoute("/api/v1/push/influx/write", http.HandlerFunc(a.handleSeriesPush), http.MethodPost)
registerer.RegisterRoute("/api/v2/write", http.HandlerFunc(a.handleSeriesPush), http.MethodPost)
registerer.RegisterRoute("/healthz", http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think Justin added healthz in a different PR (#51), let's make sure this isn't redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks

@grafanabot

This comment has been minimized.

@grafanabot

This comment has been minimized.

1 similar comment
@grafanabot

This comment has been minimized.

@grafanabot
Copy link

Go coverage report:

Click to expand.
File %
github.com/grafana/influx2cortex/pkg/errorx/errors.go 67.3%
github.com/grafana/influx2cortex/pkg/errorx/http_translator.go 100.0%
github.com/grafana/influx2cortex/pkg/influx/api.go 100.0%
github.com/grafana/influx2cortex/pkg/influx/errors.go 100.0%
github.com/grafana/influx2cortex/pkg/influx/parser.go 84.1%
github.com/grafana/influx2cortex/pkg/influx/proxy.go 63.4%
github.com/grafana/influx2cortex/pkg/influx/recorder.go 91.7%
github.com/grafana/influx2cortex/pkg/remotewrite/client.go 69.1%
github.com/grafana/influx2cortex/pkg/remotewrite/measured_client.go 88.9%
github.com/grafana/influx2cortex/pkg/remotewrite/recorder.go 100.0%
github.com/grafana/influx2cortex/pkg/route/registerer.go 100.0%
github.com/grafana/influx2cortex/pkg/server/middleware/http_auth.go 0.0%
github.com/grafana/influx2cortex/pkg/server/middleware/http_fake_auth.go 0.0%
github.com/grafana/influx2cortex/pkg/server/middleware/instrument.go 6.7%
github.com/grafana/influx2cortex/pkg/server/middleware/logging.go 69.7%
github.com/grafana/influx2cortex/pkg/server/middleware/middleware.go 0.0%
github.com/grafana/influx2cortex/pkg/server/middleware/request_limits.go 81.8%
github.com/grafana/influx2cortex/pkg/server/middleware/response.go 57.1%
github.com/grafana/influx2cortex/pkg/server/middleware/route_matcher.go 0.0%
github.com/grafana/influx2cortex/pkg/server/middleware/tracer.go 13.0%
github.com/grafana/influx2cortex/pkg/server/server.go 83.3%
github.com/grafana/influx2cortex/pkg/tenant/resolver.go 75.6%
github.com/grafana/influx2cortex/pkg/tenant/tenant.go 91.7%
github.com/grafana/influx2cortex/pkg/util/bytereplacer/byte_replacer.go 100.0%
github.com/grafana/influx2cortex/pkg/util/extract_forwarded.go 66.7%
github.com/grafana/influx2cortex/pkg/util/http.go 68.3%
github.com/grafana/influx2cortex/pkg/util/log/experimental.go 0.0%
github.com/grafana/influx2cortex/pkg/util/log/log.go 0.0%
github.com/grafana/influx2cortex/pkg/util/log/rate_limit.go 90.0%
github.com/grafana/influx2cortex/pkg/util/log/wrappers.go 0.0%
github.com/grafana/influx2cortex/pkg/util/push/push.go 60.0%
total 65.3%

Go lint report:

No issues found. 😎

Copy link
Contributor

@ywwg ywwg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is really great! I do hope we're able to pull a lot of this out into a reusable library for the next proxy

Comment on lines +32 to +40
// These tests verify that the Influx proxy is able to take in InfluxDB line protocol,
// correctly parse it and convert it into a timeseries, and write the timeseries to
// Cortex. Several services are run to execute the tests. The InfluxDB client is used to
// send line protocol to the proxy. The proxy service accepts the line protocol, parses it,
// and writes it to the Cortex service. The Prometheus client and API are used to query
// Prometheus to verify that the line protocol was parsed and converted into the expected
// timeseries, and that the timeseries was successfully written.

func TestSuite(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small tweak so that the documentation generator will pick up this comment. (we don't generate docs automatically, but we could, and this would be a nice comment to see if we do)

Suggested change
// These tests verify that the Influx proxy is able to take in InfluxDB line protocol,
// correctly parse it and convert it into a timeseries, and write the timeseries to
// Cortex. Several services are run to execute the tests. The InfluxDB client is used to
// send line protocol to the proxy. The proxy service accepts the line protocol, parses it,
// and writes it to the Cortex service. The Prometheus client and API are used to query
// Prometheus to verify that the line protocol was parsed and converted into the expected
// timeseries, and that the timeseries was successfully written.
func TestSuite(t *testing.T) {
// TestSuite verifies that the Influx proxy is able to take in InfluxDB line protocol,
// correctly parses it and converts it into a timeseries, and writes the timeseries to
// Cortex. Several services are run to execute the tests. The InfluxDB client is used to
// send line protocol to the proxy. The proxy service accepts the line protocol, parses it,
// and writes it to the Cortex service. The Prometheus client and API are used to query
// Prometheus to verify that the line protocol was parsed and converted into the expected
// timeseries, and that the timeseries was successfully written.
func TestSuite(t *testing.T) {

Copy link
Contributor

@leizor leizor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@carrieedwards carrieedwards merged commit f3c1104 into main Apr 20, 2022
@carrieedwards carrieedwards deleted the cedwards/add-acceptance-tests branch April 20, 2022 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants