Define JMX metrics "conventions"

[SylvainJuge]

The goal of this issue is to describe (in comments) all the challenges/compromises found while trying to align the JMX yaml definitions in instrumentation/jmx-metrics/library.

The current migration effort involves the following steps for each supported system:

• move the existing yaml file into library
• include metrics that are part of jmx-scraper/jmx-gatherer if needed
• try to make the metrics fit the semantic conventions
  • metric names lowercase and split with dots
  • namespace metrics per system
  • namespace metrics attributes or reuse existing semconv attributes
  • units of time duration should be in seconds
  • existing semconv attributes should be reused whenever possible

[SylvainJuge]

JVM runtime metrics #13392

• metrics are defined in semantic conventions so there wasn't any discussion needed on the metrics to capture and/or their definitions.
• some of the metrics can't be captured through JMX metrics in YAML as they rely on JMX notifications which require dedicated code (histograms). For example: jvm.gc.duration.
• some of the metrics attributes can't be captured through JMX metrics in YAML, so the metrics do not have all the recommended attributes, which is quite similar to the java8/java17 runtime versions that do not produce identical metric attributes. For example jvm.thread.count metric do not have jvm.thread.daemon or jvm.thread.state metric attributes.
• experimental metrics are included in the YAML definition, there is no config option to enable/disable them unlike the ones provides in runtime instrumentation. For example jvm.memory.init

[SylvainJuge]

Tomcat metrics: #13650

• pre-aggregated metric tomcat.request.duration.max :
  • .max as suffix
  • gauge metric type, even if it never decreases, as the value can't be aggregated
• pre-aggregated metric to represent total duration tomcat.request.duration
  • no .total suffix to fit "do not use total" semconv rule.
  • This potentially prevents later providing an histogram of each request duration with the same metric name, if we pick an .total suffix we don't have this issue.
  • This probably need further discussion.
• use distinct metrics for tomcat.thread.count and tomcat.thread.busy.count when breakdown by a metric attribute is not possible (because there is no direct reporting of the "idle thread count".