Support Tail Based Sampling Processor From OTEL Collector Extension

[mahadzaryab1]

Which problem is this PR solving?

• Closes Support tail-based sampling from OTEL Collector #5867

Description of the changes

• Added the tail-based sampling processor extension from otel to jaeger
• Added a docker compose to demonstrate usage of the tail-based sampling processor extension in jaeger.
• Added an end to end integration test to test that the new processor works as expected
• Added a README to the docker compose setup describing the setup and usage of the new processor

How was this change tested?

• An end to end integration test was added and is run from the CI

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.82%. Comparing base (9a30dfc) to head (c50ded3).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5878      +/-   ##
==========================================
- Coverage   96.83%   96.82%   -0.02%     
==========================================
  Files         342      342              
  Lines       16524    16525       +1     
==========================================
- Hits        16001    16000       -1     
- Misses        337      339       +2     
  Partials      186      186              

Flag	Coverage Δ
badger_v1	8.05% <ø> (ø)
badger_v2	1.82% <ø> (ø)
cassandra-3.x-v1	16.62% <ø> (ø)
cassandra-3.x-v2	1.75% <ø> (ø)
cassandra-4.x-v1	16.62% <ø> (ø)
cassandra-4.x-v2	1.75% <ø> (ø)
elasticsearch-6.x-v1	18.79% <ø> (+0.01%)	⬆️
elasticsearch-7.x-v1	18.85% <ø> (+0.01%)	⬆️
elasticsearch-8.x-v1	19.03% <ø> (ø)
elasticsearch-8.x-v2	1.82% <ø> (ø)
grpc_v1	9.49% <ø> (ø)
grpc_v2	7.16% <ø> (ø)
kafka-v1	9.74% <ø> (ø)
kafka-v2	1.82% <ø> (ø)
memory_v2	1.82% <ø> (ø)
opensearch-1.x-v1	18.90% <ø> (+0.01%)	⬆️
opensearch-2.x-v1	18.90% <ø> (ø)
opensearch-2.x-v2	1.82% <ø> (ø)
tailsampling-processor	0.46% <ø> (?)
unittests	95.30% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[yurishkuro]

I tested the binary sizes

• without this change - 75Mb
• with this change - 114Mb
• with only tail sampler (without load balancer) - 78Mb

I would suggest we only include tail sampler. That's the component that's most useful in the final collector, which needs to be Jaeger. But the upstream load balancing collectors can be just OTEL Collectors.

[yurishkuro]

this runs go test, but when do you start the docker compose environment?

All other e2e tests have a driver script that orchestrates all components of the test, e.g. scripts/es-integration-test.sh

[mahadzaryab1]

I'm not using the docker-compose environment for my test. Calling e2eInitialize is enough to start the Jaeger collector. You can simply run this test by calling make tail-sampling-integration-test. Let me know if you want to change any of this setup though.

[yurishkuro]

Fair enough. However, it means that the new docker compose file will begin to rot since it's not being exercised by the CI, something we tried to avoid (e.g. see e2e spm test). So it would be good to actually combine using docker compose with e2e test.

[mahadzaryab1]

Ah I see! What would you want this to look like? The current docker-compose set up generates load using tracegen which ideally we wouldn't want in the integration test so we can manually generate those. And the existing setup in the E2E tests does some nice things for us like flush the storage in between tests.

[yurishkuro]

Well, that's why I was asking from the start what your plan would be. The way you are using e2e_integration framework is very lightweight, and I could easily see an alternative setup where everything is just orchestrated from a shell script

• run docker-compose with one config
  • maybe don't include tracegen in compose, run it manually
• do a curl against query service to retrieve service names as JSON (trivial to write)
• shut down docker-compose (to clear the storage) and run again with different config

If you are interested to pursue this approach, I would suggest still merging this PR first so that we already have something in place. Can you finish the README?

[mahadzaryab1]

@yurishkuro That sounds good to me and I can pursue that approach in a follow-up PR. And yes, working on the README now. Will push it up soon.

[yurishkuro]

make sure to do git pull, I pushed updates to go.mod to resolve conflicts

[mahadzaryab1]

make sure to do git pull, I pushed updates to go.mod to resolve conflicts

Thank you so much for doing this for me!

[mahadzaryab1]

@yurishkuro the README and the rest of the PR is ready for review now

[yurishkuro]

LGTM, thanks!

[mahadzaryab1]

@yurishkuro - there looks to be a failing test in the CI. Is it a flaky test? I don't believe its related to my changes.

[yurishkuro]

🎉 🎉 🎉