Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][broker] PIP-264: Add Java runtime metrics #22616

Merged
merged 28 commits into from
May 7, 2024

Conversation

dragosvictor
Copy link
Contributor

PIP-264

Motivation

Adds support for exporting Java runtime metrics via the OpenTelemetry pipeline. For ease of implementation, relies on the built-in OTel library providing this exact functionality. We can add any extra metrics we see fit separately, as deemed necessary.

Modifications

Add and initialize field type io.opentelemetry.instrumentation.runtimemetrics.java17.RuntimeMetrics in OpenTelemetryService. This object exposes the desired runtime metrics. Placing it inside OpenTelemetryService automatically makes it available for all users of the class (broker, proxy and function workers) without any further work needed. Enable metric collection for all Java features.

Some of the metrics are presently marked as experimental (see doc): jvm.memory.init, jvm.buffer.memory.usage, jvm.buffer.memory.limit, jvm.buffer.count. Others are exposed only if flag OTEL_SEMCONV_STABILITY_IN includes value jvm. Since they are still useful, enable their collection too.

The original (Prometheus) metric list can be consulted here.

For a full description of the semantics of the OpenTelemetry metrics, see this.

OpenTelemetry (new) metric name Prometheus (old) metric name
jvm.buffer.count jvm_buffer_pool_used_buffers
jvm.buffer.memory.limit jvm_buffer_pool_capacity_bytes
jvm.buffer.memory.usage jvm_buffer_pool_used_bytes
jvm.class.count jvm_classes_currently_loaded
jvm.class.loaded jvm_classes_loaded_total
jvm.class.unloaded jvm_classes_unloaded_total
jvm.cpu.time process_cpu_seconds_total
jvm.gc.duration jvm_gc_collection_seconds
jvm.memory.committed jvm_memory_bytes_committed
jvm.memory.init jvm_memory_bytes_init
jvm.memory.limit jvm_memory_bytes_max
jvm.memory.used jvm_memory_bytes_used
jvm.memory.used_after_last_gc jvm_memory_pool_allocated_bytes_total
jvm.thread.count jvm_threads_state, jvm_threads_current and jvm_threads_daemon

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • Added unit test org.apache.pulsar.opentelemetry.OpenTelemetryServiceTest#testJvmRuntimeMetrics verifying the respective metrics are present at runtime.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency) Added OTEL runtime metrics library
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics: Added runtime metrics as described above
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: dragosvictor#18

@github-actions github-actions bot added the doc-required Your PR changes impact docs and you will update later. label Apr 29, 2024
@dragosvictor dragosvictor marked this pull request as ready for review April 29, 2024 20:12
Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari lhotari closed this May 1, 2024
@lhotari lhotari reopened this May 1, 2024
@lhotari
Copy link
Member

lhotari commented May 3, 2024

There were quite a few OOMs in that last runs. I'll close and reopen to see if they were originating from the changes made in this PR.

@lhotari lhotari closed this May 3, 2024
@lhotari lhotari reopened this May 3, 2024
pom.xml Outdated Show resolved Hide resolved
conf/pulsar_env.sh Outdated Show resolved Hide resolved
@lhotari lhotari merged commit 788b5ae into apache:master May 7, 2024
47 of 50 checks passed
@dragosvictor dragosvictor deleted the pip-264-java-runtime-metrics branch May 7, 2024 19:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-required Your PR changes impact docs and you will update later. ready-to-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants