[Feature]: Elasticsearch v8 support - LFX Mentorship

[yurishkuro]

For the application process please refer to this issue.

• Description: Jaeger has always supported Elasticsearch (ES) as an official backend. Unfortunately, the Go driver we are using, olivere/elastic, does not support ESv8 and not planning to release any new versions. Despite the licensing changes, Elasticsearch remains a popular choice and v8 support question often comes up. In this project we want to add official support for ESv8. However, we also want to take this opportunity to do a better alignment with OpenTelemetry Collector, which already has an ESv8 exporter, perhaps we can use it directly and minimize the amount of code.
• Expected Outcomes:
  • Use OTEL ESv8 exporter from Jaeger, if possible, otherwise build internal implementation
  • Stretch goal: use ES data as the source for Jaeger SPM views
• Recommended Skills:
  • Go: go.mod, bing able to understand existing code, coding, testing ,refactoring
  • Elasticsearch: understand ES architecture, shards, replicas, inserting data, searching for data, indices, index templates, index aliases, index rotations
• Mentor(s):
  • Yuri Shkuro (@yurishkuro)
• Mentee:
  • @pmuls99

Related: #3571

Resources

• https://medium.com/opentracing/take-opentracing-for-a-hotrod-ride-f6e3141f7941
• current ES implementation https://github.com/jaegertracing/jaeger/tree/main/plugin/storage/es
• https://www.jaegertracing.io/docs/1.47/deployment/#elasticsearch
• https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/elasticsearchexporter

[enesonus]

Hi @yurishkuro
I am planning to apply for this project for the CNCF mentoring program. Are there any resource recommendations you have or approaches that the Jaeger team follows so that I can learn more about the subject and help better? I have some knowledge on Go but did not worked much with Elasticsearch before.

[yurishkuro]

@enesonus it's a good question. Long time ago I read some book on ES, it explained it pretty well, but I don't remember which book, and it's outdated by now. I see lots of tutorials and courses online (some are not free). If you find one that it useful, feel free to post here. In order to be successful in this project one needs to understand the elastic write & query APIs, how it builds indices, etc.

[wasup-yash]

I find this project interesting as it implements the new feature and adds the support for v8 functionality, Looking forward to submitting the Proposal for the same as a part of the mentorship program.

[kanha-gupta]

@yurishkuro API creation is a prominent thing to do in this project right ?

[Sayantan-Ch]

Hi @yurishkuro
Came across this project from CNFC mentorship portal. This is an interesting project to work on. I have some experience of building reverse index based search engine using Apache Lucene ( the native framework behind ES). But I don't have a lot of working idea with Jaegar. Could you point me towards some resources which can help me get upto speed.

[yurishkuro]

@Sayantan-Ch I added some links to the description

[tanayvaswani]

Thanks @yurishkuro, the resources are really helpful.

[anirudhprabhakaran3]

Hello! just wanted to register my interest for this project. Will be going through the provided references and making a proposal, thanks!

[roopeshsn]

@enesonus it's a good question. Long time ago I read some book on ES, it explained it pretty well, but I don't remember which book, and it's outdated by now. I see lots of tutorials and courses online (some are not free). If you find one that it useful, feel free to post here. In order to be successful in this project one needs to understand the elastic write & query APIs, how it builds indices, etc.

This is one of the good resources for understanding indices, and related terminologies that I came across: https://www.youtube.com/live/2WJFMYAri_8?feature=share

[enesonus]

Hi everyone I created this Jaeger Demo that uses Elasticsearch as a backend in order to understand how does the system work. If you are interested in please have a look!

BTW thanks for the resource, @roopeshsn it was very helpful in learning elasticsearch. I am planning to add SPM to my demo and I think this resource will be of help.

Note: I intend to continue improving the demo so if you have any suggestions please contact me at me@enesonus.com.