Add support for tracing with OpenTelemetry API

[SylvainJuge]

Description

Adding OpenTelemetry tracing to Elasticsearch Java client could provide better tracing data than what is currently captured through APM agents, which could help end-users to better understand the interactions between their application and Elasticsearch.

The current approach used in APM agents is limited to low-level details close the the HTTP request and can't provide an high-level usage patterns.

• Elastic Java agent instruments the REST client https://github.com/elastic/apm-agent-java/tree/main/apm-agent-plugins/apm-es-restclient-plugin
• OpenTelemetry instrumentation relies on REST client and Transport client : https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/elasticsearch

As a consequence:

• the captured spans naming and structure do not reflect usage of the high-level API but the low-level HTTP API
• the captured spans cardinality can become problematic as we have to guess usage patterns from URLs.
• instrumentation agents have to implement heuristics and parsing to know when it's appropriate to capture request body (for example elasticsearch_capture_request_body option)

Also, because it is a vendor-agnostic API, it allows to leverage this instrumentation for use-cases beyond existing APM tools, for example by enabling end-users to trace their high-level usage patterns into log files or a backend of their choice.

As a first step proposal, I would suggest doing the following:

• wrap high-level client usages in OTel spans, picking names that fits the entry-point in ES client.
• keep the created ES high-level client spans intermediate spans (not "exit spans"), so we can still rely on low-level instrumentation for now.

[Mpdreamz]

Not advocating for a different approach but sharing as an FYI

The javascript client:

https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/observability.html

and the .NET client:

https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/diagnostic-source.html

Take a more event based approach to observability that their respective agents leverage to create instrumentation. This also includes insights into fallover (de)serialization and node selection decisions.

This could also be considered for the java ES client if taking an extra dependency is controversial.

[SylvainJuge]

Fixed by #588 and documented in #658