Favour Tracer API above ElasticApmTracer API in plugins and add additional seperation

[raphw]

Is your feature request related to a problem?

I would like to piggyback on Elastic's agent for non-APM-tracing, but at various points, the agent is bound to implementation details of Elastic's agent. This is mainly with regards to the use of ElasticApmTracer being requested from various places. However, this does not seem necessary for the most parts, the method missing from the Tracer API are getConfig() and getObjectPoolFactory() with few exceptions.

Describe the solution you'd like

By adding only a few methods to the tracer API, the direct bounds to Elastic's APM can be removed from almost all plugins. This way it becomes possible to implement a custom agent whilst still using the comprehensive infrastructure that is offered by the Elastic agent and its plugins.

Describe alternatives you've considered

Forking the repository and implementing a copy with individual customization, but that is of course not a good long-term strategy and hinders collaboration on the main repository.

Additional context

Open Telementry offers a generic agent for implementing a blue print agent. Unfortunately, the agent does not support as many plugins as the Elastic agent and is not equally well maintained. It would be great if the Elastic agent could evolve in a direction where it would be possible to use it for other tracing purposes. As an ideal result, this would trigger additional contributions on the long run and help the Elastic agent to grow its standing in the field. To begin with, I would want to make some binary-compatible changes that improve the pluggability, but at some point one could even better differentiate the plugin SPI from the core agent API.

Implementation steps

While this change does not require many changes, it is a rather comprehensive remodularization and refactoring. It should therefore be broken up into multiple steps. In the first steps, the new API would be established. In all subsequent steps, the new API would be taken into use, thus carving out what dependencies still remain upon the agent's core. Each change thereafter would address such a step by adding additional API and further reduce the reliance on core API. All changes are based upon a proof of concept that demonstrates that such API refactoring is feasible.

Suggested steps:

1. Introduce tracer API within isolated module. #3043
2. Move global tracer and substitute agent-core types with API types where this is trivially possible. #3054
3. Reduce reliance on core API.
   3a. Avoid use of trace context static methods in favor of Tracer-defined behavior #3082
   3b. Extract configuration from core module to API #3122
   3c. Factor out reference counting to explicit API #3081
   3d. Move plugin-related utilities and API to SDK/tracer #3166
   3e. Make most plugins depend on only the SDK and tracer dependencies. #3231
   3f. Moves ServiceInfo to tracer module and reduce dependency exposure to core module of plugins. #3272
   3g. Add error reporting API to tracer-api. #3273
   3h. Introduce Tracer subtype that supports explicit span creation as required by API plugin, open tracing plugin and open telementry plugin. After an implementation attempt, I do no longer think that explicit tracing bridges should be ported to the generic API as it would require a massive extension of that API. It can be assumed that other tracers that want to use the plugins of this tracer are not interested in such bridges or offer bridges of their own, as such bridges would be double indirections.
   3i. Introduce Tracer API for log capture. #3515
   3j Introduce API for metrics collection that does not depend on agent-core #3522
   3k. Add API for instrumenting FAAS and adjust the AWS plugin #3516
4. Remove unused API and resolve naming ambiguities #3561

[raphw]

I created a repo where I tried a POV to see if this is feasible and it seems like a small change which already separates the APIs much more. raphw@53e70e0