ADOT guidelines for testing and marking partner exporters as stable

[mraditya]

ADOT guidelines for testing and marking partner exporters as stable

Objectives

1. Ensure all partner components including receivers and exporters for partner end-points are tested and verified with the latest version of the Collector (v 0.30.0) which will guarantee tracing stability in OpenTelemetry.
2. Testing includes:
   1. ADOT end-to-end testing with public service end-point
   2. ADOT Soak testing to ensure there are no CPU, memory leaks
   3. Performance testing to ensure all known thresholds/limitations are tested against and verified for these components
   4. Security testing identifying any known vulnerabilities

How to do conduct testing

1. Use the latest version of the Collector (corresponding to the tracing stable release targeting launch on 8/13)
2. Use the ADOT test framework to run end-to-end tests and soak tests. See ADOT Testing Framework guidance here.
3. Performance testing has to be done for each component based on individual service providers (vendors / partner service end point limitations)
4. Security testing includes running GoSec for Go modules; CodeQL security scans should be run and list of any known vulnerabilities should be submitted in the issue
5. Use guidelines itemized in the readme to publish end-to-end test and soak test results. See the ADOT test framework here.
6. OTEL Stable Readiness Issue: File an issue here to indicate that the partner has tested, verified and requested their components to be marked stable for the tracing stable release.
7. ADOT Stable Readiness issue: File an issue here to publish all ADOT test results for each partner component.

Expected results

1. Steps 1-4 need to be run and pass all testing success criteria.

Deliverables

1. 8/13 OTEL Collector core will be marked as stable
2. 8/23 OTEL will announce tracing stability
3. 8/20 OTEL Collector contrib components will be ready / marked stable to be included in Collector contrib release (includes AWS and partners exporters)
4. 9/23 ADOT announces tracing stable GA with X-Ray, OTLP, and Partner tracing exporters

[alolita]

The latest Collector RC as of August 2 2021 can be found at https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.31.0

[tigrannajaryan]

@alolita from what I see aws-otel-collector is still depending on Otel Collector v0.30.0. Do you plan to update to v0.31.0?
I believe once you do that the ADOT Testing Framework will run automatically and the results you are looking for will be in the Github actions, so you already did most of the hard work for all partner components, test steps 1-3 will be completed automatically.

For step 4:
a) CodeQL is run for all collector-contrib repo automatically, so provided that this passes we OK, right?
b) Similarly, if I am not wrong Gosec is also part of the linting on collector-contrib repo, so all partner components already pass this.

So, just to confirm, provided that the existing checks and tests that are in place pass, then that confirms the partner components are good and satisfy the requirements for steps 1-4. Is this correct or there is something additional you need?

[rmfitzpatrick]

@mraditya, @alolita any chance you'd be able to confirm that you will be adopting 0.31.0 in the aws-otel-collector project so that we're able to confirm all existing tests are successful for provided components in your CI env (~aws-observability/aws-otel-collector#600)? I'd be happy to contribute a PR for this that takes care of the required proto changes if desired.

[cartermp]

ACK on needing v0.31.0 - we will continue testing once it is available: https://github.com/honeycombio/telemetry-team/issues/69#issuecomment-896139534

[alolita]

Update on the Collector release and testing timelines:

Deliverables:

• 8/23 OTEL Collector core will be marked as stable
• 8/27 OTEL will announce tracing stability
• 8/27 OTEL Collector contrib components will be ready / marked stable to be included in Collector contrib release (includes AWS and partners exporters)
• 9/23 ADOT announces tracing stable GA with X-Ray, OTLP, and Partner tracing exporters

[alolita]

For partners using OTLP exporters to connect with their service end-points, please follow the steps below:

Testing includes:

• Use ADOT Collector to complete end-to-end testing with your service end-point which uses the OTLP protocol
• We are assuming that you are using your testing pipelines to test the ADOT Collector with your service end-point to ensure there are no performance degradations and security vulnerabilities.

How to do conduct testing

• Use the latest version of the Collector (corresponding to the tracing stable release which will be 0.33 as of 8/16)

Expected Result

• ADOT Stable Readiness issue: File an issue here to publish your test results for your end-to-end tests with OTLP <-> your service end-point.

Deliverables

• 8/23 OTEL Collector core will be marked as stable
• 8/27 OTEL will announce tracing stability
• 8/27 OTEL Collector contrib components will be ready / marked stable to be included in Collector contrib release (includes AWS and partners exporters)
• 9/23 ADOT announces tracing stable GA with X-Ray, OTLP, and Partner tracing exporters

[MikeGoldsmith]

@alolita please can you confirm for OTLP exporter partners exactly which tests you'd like us to run, and steps to execute them?

I've looked through the aws-collector and aws-collector-test-framework projects and am unsure what is expected for us to run. I have setup the test-framework locally and am able to execute the otlp_mock test case, but if I override the otconfig.tpl otlp_trace test case to export to Honeycomb, the test fails because the validator is unable to validate trace data was successfully processed.

Thanks

[MikeGoldsmith]

@alolita @mraditya any update on the above?

[alolita]

Hi @MikeGoldsmith Please use the latest version of the upstream collector-contrib release to run your end-to-end tests for your exporter for traces. For example, if you are using the Honeycomb end-point, test Collector ingestion (OTLP) and export to Honeycomb's service endpoint.Hope that clarifies.