Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add E2E receiver/export correctness tests #652

Closed
6 tasks
tigrannajaryan opened this issue Mar 18, 2020 · 9 comments
Closed
6 tasks

Add E2E receiver/export correctness tests #652

tigrannajaryan opened this issue Mar 18, 2020 · 9 comments
Assignees
@pmcollins
Labels
help wanted Good issue for contributors to OpenTelemetry Service to pick up
Projects
Collector
Milestone
Backlog

Comments

@tigrannajaryan
Copy link
Member

tigrannajaryan commented Mar 18, 2020 •
edited
Loading

We currently have E2E tests that benchmark the performance of various formats.

We also need E2E tests that verify the correctness of the Collector operation as it receives and exports the data in various formats. Performance tests don't verify this today. We need separate tests that will send telemetry data to the Collector, covering all possible variety of such data and then verify that the Collector exports this data precisely as it is supposed to be represented in the configured export format.

The preference is to have a matrix test that verifies many receiver/exporter combinations and uses golden data sets for verification.

Possible approach:

  • Implement a span generator that accepts several boolean, enum and numeric flags that control what kind of span to generate: with or without a particular field, how many attributes, what type of attributes, etc. Make sure to include ability to generate spans with nil fields, zero-sized slices, etc - ensure edge cases are covered.
  • Write a test that generates a variety of spans. Possibly try toggling true/false every boolean flag that the generator accepts, use all values for enum flags and use counts of 0, 1 and random higher number for numeric flags (e.g. number of attributes). Send the span via testbed, receive and compare it to the original.
  • Perform the test for all combination of receivers and exporters that are supported in the testbed (N*N tests total).
  • Make sure cases like empty spans, or empty batches of spans are covered.
  • Make the test configurable and have it accept a list of receivers and exporters to test and a list of processors to enable during the test. Make sure all default recommended processors are enabled: memorylimiter, batch, queue.
  • Export the test as a public ScenarioTraceTranslation and also call it in Contrib to test receivers and exporters in Contrib. Enable contrib processors (e.g. k8s processor).
@tigrannajaryan tigrannajaryan added the help wanted Good issue for contributors to OpenTelemetry Service to pick up label Mar 18, 2020
@mat-rumian
Copy link
Contributor

I will be happy to help with this :)

@kbrockhoff
Copy link
Member

I will soon be submitting a PR for generating and managing "Golden Data". It will have the following components and process steps:

  1. Variation parameters - Various data fields which can vary in different observations. For example for trace spans, I currently am using: Parent, Tracestate, Kind, Attributes, Events, Links, Status
  2. PICT input files - Definitions to feed the Pairwise Independent Combinatorial Testing tool PICT
  3. PICT output files - Output from Pairwise Independent Combinatorial Testing tool with recommended data combinations
  4. Golden data generator - Reads PICT output files and generates corresponding real world like data and then serializes as OTLP to files
  5. Additional test data directory - Holds additional OTLP serialized data examples not covered by the Golden Data generator
  6. Bad data recording processor - OT Collector processor to OTLP serialize data items which cause exporters and other processors to return invalid data errors. These can then be added to the additional test data directory to easily reproduce the errors.
  7. Correctness test executor - Spins up various otelcol pipeline configurations based on the appropriate PICT output file and then feeds all of the serialized data examples through the pipeline and checks the output.

@tigrannajaryan
Copy link
Member Author

@kbrockhoff great, this will be a very useful addition. Please make smaller incremental PRs if possible to make reviewing easier.

@kbrockhoff kbrockhoff mentioned this issue May 13, 2020
Merged
@tigrannajaryan tigrannajaryan added this to To do in Collector May 19, 2020
tigrannajaryan pushed a commit that referenced this issue May 26, 2020
@kbrockhoff
Generate "Golden Data" trace spans (#967)
ffd4838 
1. PICT tool input files for trace data and resulting output files from running pict from the command line. These are in `internal/goldendataset/testdata`. See: [Pairwise Independent Combinatorial Testing](https://github.com/microsoft/pict)
2. Generator for fully-populated OLTP ResourceSpans from PICT output files.

This has all the intended functionality for this PR. It has no impact on other functionality so there should be no problem in merging to master. I will follow up with other PRs to complete full "Golden Dataset" testing functionality.

**Link to tracking Issue:** #652 

**Testing:** Unit tests exist for all functionality which has been coded so far

**Documentation:** None yet. Will do so after golden dataset generation coding is complete.
@kbrockhoff kbrockhoff mentioned this issue Jun 1, 2020
Merged
@tigrannajaryan tigrannajaryan added this to the Beta 0.4 milestone Jun 3, 2020
@kbrockhoff
Copy link
Member

I plan to write these tests to verify the API in the generalize-testbed branch. If @pmcollins has not started, you can assign the ticket to me. Or else I am happy to advise on how to write the tests using the generalized testbed.

@pmcollins
Copy link
Member

Either way works for me, @kbrockhoff . I was part way through a proof of concept for how to test for correctness: two pipelines, a pipeline under test and a test harness pipeline. The test harness pipeline has a processor that sends metrics to an exporter that is configured to talk to the pipeline under test, from which it is configured to also receive metrics. The same processor in the test harness pipeline compares the received metrics to what it sent. But maybe the generalize-testbed branch is the way to go instead (I was mostly out last week and wasn't aware of it).

Originally, I think we, including @tigrannajaryan, thought that maybe you (or someone) could work on the traces tests and I could work on the metrics tests (I'm more familiar with metrics). But maybe the way to go is for one of us to hold off until the other has an implementation. I'm happy to be the one to hold off since it looks like you have made significantly more progress than I have.

@tigrannajaryan
Copy link
Member Author

tigrannajaryan commented Jun 9, 2020 •
edited
Loading

I think since @kbrockhoff started the trace PICT generator it is best that he continues working on it and @pmcollins you can work on the similar capability and tests for metrics part. Pablo, you are right that it may be best to wait a bit until Kevin is done with testbed refactoring. Kevin, do you need more changes to the testbed after this PR is merged?

@kbrockhoff
Copy link
Member

I was planning to do all the refactoring in one PR. Still have a few improvements to make yet before it is ready for merging.

@flands flands modified the milestones: Beta 0.4, Beta 0.5 Jun 16, 2020
tigrannajaryan pushed a commit that referenced this issue Jun 17, 2020
@kbrockhoff
Generalize testbed and add trace translation correctness tests (#1062)
e3df9f3 
Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use.

**Link to tracking Issue:** Provides the support to cleanly implement #652, #1022, #1023, #1027, #1031

**Testing:** All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations.

**Documentation:** Godocs on all public methods.
@kbrockhoff
Copy link
Member

Testbed changes have been merged to master. Correctness tests for traces have been completed as part of the PR. Work on metrics correctness tests can now proceed.

@flands flands modified the milestones: Beta 0.5.0, Beta 0.5.1 Jul 6, 2020
@tigrannajaryan tigrannajaryan moved this from To do to Review in progress in Collector Jul 6, 2020
wyTrivail pushed a commit to mxiamxia/opentelemetry-collector that referenced this issue Jul 13, 2020
@kbrockhoff @wyTrivail
Generate "Golden Data" trace spans (open-telemetry#967)
945e7b0 
1. PICT tool input files for trace data and resulting output files from running pict from the command line. These are in `internal/goldendataset/testdata`. See: [Pairwise Independent Combinatorial Testing](https://github.com/microsoft/pict)
2. Generator for fully-populated OLTP ResourceSpans from PICT output files.

This has all the intended functionality for this PR. It has no impact on other functionality so there should be no problem in merging to master. I will follow up with other PRs to complete full "Golden Dataset" testing functionality.

**Link to tracking Issue:** open-telemetry#652 

**Testing:** Unit tests exist for all functionality which has been coded so far

**Documentation:** None yet. Will do so after golden dataset generation coding is complete.
wyTrivail pushed a commit to mxiamxia/opentelemetry-collector that referenced this issue Jul 13, 2020
@kbrockhoff @wyTrivail
Generalize testbed and add trace translation correctness tests (open-…
874ade0 
…telemetry#1062)

Extracted out TestResultsSummary (in testbed/testbed/results.go), DataProvider (in testbed/testbed/data_provider.go), OtelcolRunner (in testbed/testbed/otelcol_runner.go), TestCaseValidator (in testbed/testbed/validator.go) interfaces with multiple implementations. Added tracing correctness tests in testbed/correctness using the testbed with different implementations of the 5 interfaces listed than what the perf tests use.

**Link to tracking Issue:** Provides the support to cleanly implement open-telemetry#652, open-telemetry#1022, open-telemetry#1023, open-telemetry#1027, open-telemetry#1031

**Testing:** All existing testbed-driven tests still pass. Correctness tests run without any panics. Correctness tests are reporting a number of bugs with translations.

**Documentation:** Godocs on all public methods.
@flands flands modified the milestones: Beta 0.6.0, Beta 0.7.0 Jul 15, 2020
This was referenced Jul 20, 2020
Merged
Merged
Merged
@pmcollins pmcollins mentioned this issue Jul 28, 2020
Merged
@bogdandrutu bogdandrutu removed this from the Beta 0.7.0 milestone Jul 30, 2020
@bogdandrutu bogdandrutu added this to the Beta 0.8.0 milestone Jul 30, 2020
@pmcollins pmcollins mentioned this issue Aug 5, 2020
Merged
bogdandrutu pushed a commit that referenced this issue Aug 5, 2020
@pmcollins
Move some content from correctness_test.go to utils.go (#1497)
56a22ad 
* Move some content from correctness_test.go to utils.go

This change makes these functions/types available from the metrics
package, where they will be needed to address issue #652.

* Add comments for exported types and fcns

* Address PR comments

* Fix lint
@bogdandrutu bogdandrutu modified the milestones: Beta 0.8.0, Beta 0.9.0 Aug 12, 2020
This was referenced Aug 20, 2020
Closed
Merged
This was referenced Sep 1, 2020
Merged
Merged
@tigrannajaryan
Copy link
Member Author

Closing this issue, correctness tests now exist.

Collector automation moved this from Review in progress to Done Sep 16, 2020
hughesjj pushed a commit to hughesjj/opentelemetry-collector that referenced this issue Apr 27, 2023
@jeffreyc-splunk
Add SAST/OSS scans to gitlab ci (open-telemetry#652)
c9fdbc3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pmcollins pmcollins

Labels
help wanted Good issue for contributors to OpenTelemetry Service to pick up
Projects
No open projects
Collector
  
Done
Milestone
Backlog
Development

No branches or pull requests
6 participants
@pmcollins @flands @bogdandrutu @kbrockhoff @tigrannajaryan @mat-rumian