[feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry

[asafm]

PIP: #21709

Motivation

As explained in the PIP, the motivation is to allow users of Pulsar to set an alert when the actual age of the oldest unacknowledged message for a topic is nearing its defined backlog quota time limit.

Modifications

• Four fields related to backlog telemetry were added to TopicStats per the PIP
• Four metrics related to backlog telemetry added per the PIP
• Cleaned up roughly 25 IntelliJ warnings
• A new auxiliary class named PrometheusMetricsClient was introduced, making retrieving and querying Prometheus metrics easier. The primary parsing method was refactored out of PrometheusMetricsTest as at least 7 test classes used this parse method directly, which didn't seem appropriate.
• Javadoc of Consumer.seek() was enhanced as it lacked an important distinction to the effect of it. Added test to verify.
• Added log4j.xml to pulsar-broker/src/test/resources, making it easy for developers to change the log level when running unit tests. Before they copy-paste a file into it and delete it.

Verifying this change

• Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: asafm#4

[github-actions]

@asafm Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[lhotari]

LGTM. Impressive work @asafm!

[dragosvictor]

Looks good, noted a couple of ideas below.

[asafm]

Thanks for the comments, @dragosvictor, very helpful! I replied to all. Can you re-review and resolve if it is solved?

[dragosvictor]

Thanks for following up @asafm, looks good to me!

[codecov-commenter]

Codecov Report

Attention: 34 lines in your changes are missing coverage. Please review.

Comparison is base (529e1ab) 73.69% compared to head (8953ffa) 73.59%.
Report is 26 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21816      +/-   ##
============================================
- Coverage     73.69%   73.59%   -0.11%     
+ Complexity    32352    32343       -9     
============================================
  Files          1858     1860       +2     
  Lines        138144   138396     +252     
  Branches      15141    15175      +34     
============================================
+ Hits         101812   101855      +43     
- Misses        28490    28646     +156     
- Partials       7842     7895      +53     

Flag	Coverage Δ
inttests	24.15% <19.14%> (-0.11%)	⬇️
systests	23.96% <19.14%> (+0.19%)	⬆️
unittests	72.86% <85.53%> (-0.13%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...che/bookkeeper/mledger/impl/ManagedLedgerImpl.java	80.49% <100.00%> (-0.68%)	⬇️
...ookkeeper/mledger/impl/ManagedLedgerMBeanImpl.java	89.70% <100.00%> (+0.39%)	⬆️
...rg/apache/pulsar/broker/service/BrokerService.java	81.14% <100.00%> (-0.01%)	⬇️
...n/java/org/apache/pulsar/broker/service/Topic.java	36.36% <ø> (ø)
...ker/service/persistent/PersistentTopicMetrics.java	100.00% <100.00%> (ø)
...broker/stats/prometheus/AggregatedBrokerStats.java	100.00% <100.00%> (ø)
...ker/stats/prometheus/AggregatedNamespaceStats.java	100.00% <100.00%> (ø)
...ker/stats/prometheus/NamespaceStatsAggregator.java	90.40% <100.00%> (+0.62%)	⬆️
...che/pulsar/broker/stats/prometheus/TopicStats.java	94.19% <100.00%> (+0.21%)	⬆️
...oker/service/nonpersistent/NonPersistentTopic.java	71.14% <50.00%> (-0.12%)	⬇️
... and 5 more

... and 81 files with indirect coverage changes

[codelipenghui]

I will continue to review the remaining 2 classes.

• PersistentTopic.java
• BacklogQuotaManagerTest.java

[codelipenghui]

Sorry, which test covers the cached timeBasedBacklogQuotaCheckResult really works?

[asafm]

@dragosvictor Can I resolve your comments?

[dragosvictor]

@asafm Yes, for some reason I don't have access to resolve my own comments.

[lhotari]

@codelipenghui This is a large change and causes a lot of merge conflicts. Do we really want this in maintenance branches?

[codelipenghui]

@lhotari Yes, so I choose to create PR to cherry-pick it. So that I can get more eyes for review. The reason is the feature is existing for long time, but users don't have way to move it production due to the missed metrics(alerts).

[lhotari]

@codelipenghui Do you also have a chance to cherry-pick this to branch-3.2 ? there are quite a few merge conflicts.