[improve][monitor] Rename "GC Pauses" to "GC Time" in Pulsar JVM dashboards

[lhotari]

Motivation

• source metric is not about GC pauses.
• the source metric is "jvm_gc_collection_seconds_sum" which is not the same as "GC pause"
• The metric is defined in Prometheus Java client
  • "Time spent in a given JVM garbage collector in seconds."
  • source code is https://github.com/prometheus/client_java/blob/e68daf23336eb5de7856df406eb1d497f51ad3be/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot/GarbageCollectorExports.java#L53
    • calls GarbageCollectorMXBean.getCollectionTime() - Javadoc https://docs.oracle.com/en/java/javase/11/docs/api/java.management/java/lang/management/GarbageCollectorMXBean.html#getCollectionTime() "Returns the approximate accumulated collection elapsed time in milliseconds."
    • Stackoverflow Q&A: https://stackoverflow.com/a/44686539

Modifications

Replace "GC Pauses" with "GC Time" in Grafana JVM dashboards.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

There's also a metric for GC Pauses in Pulsar. The metric name is jvm_full_gc_pause.
Here's the implementation:

pulsar/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java

Lines 93 to 94 in 52d8fe0

	metrics.put("jvm_full_gc_pause", currentFullGcTime);
	metrics.put("jvm_full_gc_count", currentFullGcCount);

It uses total safepoint time metric under the covers,

pulsar/pulsar-common/src/main/java/org/apache/pulsar/common/stats/JvmDefaultGCMetricsLogger.java

Line 124 in 4dcb166

long newSafePointTime = getTotalSafepointTime();

It's a bit misleading to call it "full gc" since it's more than full gc. More details in https://blanco.io/blog/jvm-safepoint-pauses/

[codecov-commenter]

Codecov Report

Merging #18891 (61c209c) into master (3180a4a) will decrease coverage by 2.85%.
The diff coverage is 33.33%.

@@             Coverage Diff              @@
##             master   #18891      +/-   ##
============================================
- Coverage     46.17%   43.31%   -2.86%     
+ Complexity    10359     8404    -1955     
============================================
  Files           703      597     -106     
  Lines         68845    56863   -11982     
  Branches       7382     5902    -1480     
============================================
- Hits          31788    24632    -7156     
+ Misses        33448    29422    -4026     
+ Partials       3609     2809     -800     

Flag	Coverage Δ
unittests	43.31% <33.33%> (-2.86%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../main/java/org/apache/pulsar/PulsarStandalone.java	0.00% <0.00%> (ø)
...broker/intercept/ManagedLedgerInterceptorImpl.java	0.00% <0.00%> (ø)
...va/org/apache/pulsar/client/impl/ConsumerImpl.java	15.09% <0.00%> (-0.04%)	⬇️
...he/pulsar/client/impl/MultiTopicsConsumerImpl.java	22.78% <0.00%> (-0.09%)	⬇️
...nsaction/pendingack/impl/PendingAckHandleImpl.java	36.61% <50.00%> (-15.23%)	⬇️
.../pulsar/broker/service/AbstractBaseDispatcher.java	49.71% <66.66%> (-7.76%)	⬇️
...sistent/PersistentDispatcherMultipleConsumers.java	47.56% <100.00%> (-4.17%)	⬇️
...g/apache/pulsar/client/impl/RawBatchConverter.java	3.12% <0.00%> (-89.07%)	⬇️
...g/apache/pulsar/broker/service/StreamingStats.java	0.00% <0.00%> (-83.79%)	⬇️
...sar/broker/stats/metrics/ManagedLedgerMetrics.java	23.88% <0.00%> (-76.12%)	⬇️
... and 183 more

[tisonkun]

LGTM. Thanks!