Add subscription backlog size info for topicstats. #9302

MarvinCai · 2021-01-25T00:06:47Z

Motivation

Add ability to fetch backlog size for subscription, add flag in topic-stats partitioned-topic-stats for getting backlog size for subscriptions.
Sample output

./pulsar-admin topics partitioned-stats zxc-t/zxc-ns/zxc-p -gpb -sbs
{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "bytesInCounter" : 0,
  "msgInCounter" : 0,
  "bytesOutCounter" : 0,
  "msgOutCounter" : 0,
  "averageMsgSize" : 0.0,
  "msgChunkPublished" : false,
  "storageSize" : 875,
  "backlogSize" : 585,
  "publishers" : [ ],
  "waitingPublishers" : 0,
  "subscriptions" : {
    "zxc-sub-1" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "bytesOutCounter" : 0,
      "msgOutCounter" : 0,
      "msgRateRedeliver" : 0.0,
      "chuckedMessageRate" : 0,
      "msgBacklog" : 10,
      "backlogSize" : 585,
      "msgBacklogNoDelayed" : 10,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "msgRateExpired" : 0.0,
      "totalMsgExpired" : 0,
      "lastExpireTimestamp" : 0,
      "lastConsumedFlowTimestamp" : 0,
      "lastConsumedTimestamp" : 0,
      "lastAckedTimestamp" : 0,
      "lastMarkDeleteAdvancedTimestamp" : 0,
      "consumers" : [ ],
      "isDurable" : true,
      "isReplicated" : false,
      "consumersAfterMarkDeletePosition" : { },
      "nonContiguousDeletedMessagesRanges" : 0,
      "nonContiguousDeletedMessagesRangesSerializedSize" : 0
    },
    "zxc-sub-2" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "bytesOutCounter" : 0,
      "msgOutCounter" : 0,
      "msgRateRedeliver" : 0.0,
      "chuckedMessageRate" : 0,
      "msgBacklog" : 5,
      "backlogSize" : 295,
      "msgBacklogNoDelayed" : 5,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "msgRateExpired" : 0.0,
      "totalMsgExpired" : 0,
      "lastExpireTimestamp" : 0,
      "lastConsumedFlowTimestamp" : 0,
      "lastConsumedTimestamp" : 0,
      "lastAckedTimestamp" : 0,
      "lastMarkDeleteAdvancedTimestamp" : 0,
      "consumers" : [ ],
      "isDurable" : true,
      "isReplicated" : false,
      "consumersAfterMarkDeletePosition" : { },
      "nonContiguousDeletedMessagesRanges" : 0,
      "nonContiguousDeletedMessagesRangesSerializedSize" : 0
    }
  },
  "replication" : { },
  "nonContiguousDeletedMessagesRanges" : 0,
  "nonContiguousDeletedMessagesRangesSerializedSize" : 0,
  "metadata" : {
    "partitions" : 5
  },
  "partitions" : { }
}

Modifications

In ManagedLedgerImpl add API to get backlog size starting from specific position.
In Admin Rest API and CLI add option to get subscription backlog.

Verifying this change

Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
Added check in existing test to verify backlog size.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API: yes (added option to get subscription's backlog size when getting topic-stats/partitioned-topic-stats)
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: no
The admin cli options: yes (added option to get subscription's backlog size when getting topic-stats/partitioned-topic-stats)
Anything that affects deployment: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs / JavaDocs

zymap · 2021-01-25T00:30:25Z

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

+            doc = "Enable expose the backlog size fro each subscription when generating stats.\n" +
+                    " Locking is used for fetching the status so default to false."
+    )
+    private boolean exposeSubscriptionBacklokSizeInPrometheus = false;


Suggested change

private boolean exposeSubscriptionBacklokSizeInPrometheus = false;

private boolean exposeSubscriptionBacklogSizeInPrometheus = false;

I think we don't need to introduce a new configuration for this metric? If users enabled the exposeTopicLevelMetricsInPrometheus, this metric should be exposed? If it will bring a lot of expenses, we should consider adding a new configuration such as exposePreciseBacklogInPrometheus. What do you think @zymap @MarvinCai?

@codelipenghui
I think it is actually quiet expensive, it's using existing estimateBacklogFromPosition(PositionImpl pos) to get the backlog size, which requires locking the whole MLedger object. And this will happen for every subscription, periodic metric generation with this subscription backlog size could affect performance and could potentially compete with the ledger trimming task, so I think we should add a flag for it and disable by default.

zymap · 2021-01-25T00:36:11Z

pulsar-client-admin/src/main/java/org/apache/pulsar/client/admin/Topics.java

@@ -764,10 +766,11 @@ default void delete(String topic, boolean force) throws PulsarAdminException {
     * @throws PulsarAdminException
     *             Unexpected error
     */
-    TopicStats getStats(String topic, boolean getPreciseBacklog) throws PulsarAdminException;
+    TopicStats getStats(String topic, boolean getPreciseBacklog,


I think we shouldn't change the original API, that's will bring the compatibility issue with the PulsarAdmin. Let's create a new API for this.

@zymap
From what I've seen, this is only used by Admin CLI (CmdTopics.java), which provides default value . for all needed parameters, e.g. getPreciseBacklog and subscriptionBacklogSize. And there's another default implementation of TopicStats getStats(String topic) which is used in some other places, it also has default value provided for getPreciseBacklog and subscriptionBacklogSize.
So it shouldn't create compatibility issue for users.

I am thinking this is a user-faced API, user can call it from the PulsarAdmin. This change might be breaking their code.

Make sense, I'll fix it.

@zymap added a default default TopicStats getStats(String topic, boolean getPreciseBacklog) to make it backward compatible. And existing customer rely on this API won't be affected.

MarvinCai · 2021-01-28T01:09:12Z

@zymap can you take another look

zymap

Overall looks good to me.

MarvinCai · 2021-01-28T17:04:44Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-28T23:43:36Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-29T17:40:32Z

/pulsarbot run-failure-checks

sijie · 2021-01-30T00:35:08Z

@MarvinCai Can you rebase it?

…-backlogsize

MarvinCai · 2021-01-30T04:44:58Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-30T07:50:48Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-30T09:43:04Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-30T18:05:40Z

/pulsarbot run-failure-checks

…-backlogsize

MarvinCai · 2021-01-31T02:46:26Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-31T07:42:11Z

/pulsarbot run-failure-checks

MarvinCai · 2021-01-31T19:36:42Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T06:35:25Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T16:50:33Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T20:20:09Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T21:16:42Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T22:03:29Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-01T22:34:54Z

@sijie done, rebased upon newer version master and resolved conflict.

zymap · 2021-02-02T00:41:33Z

@MarvinCai There has a new conflicting file need to resolve...

…-backlogsize

MarvinCai · 2021-02-02T04:32:58Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-02T17:39:07Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-03T04:46:03Z

/pulsarbot run-failure-checks

MarvinCai · 2021-02-03T18:19:00Z

/pulsarbot run-failure-checks

zymap · 2021-02-04T02:17:37Z

There has something wrong with the SQL tests and @gaoran10 is working on it.

…-backlogsize

MarvinCai · 2021-02-06T00:37:05Z

/pulsarbot run-failure-checks

gaoran10

LGTM

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

…-backlogsize

MarvinCai · 2021-02-07T08:09:28Z

/pulsarbot run-failure-checks

lhotari · 2021-02-09T17:47:56Z

pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdTopics.java

        @Override
        void run() throws Exception {
            String topic = validateTopicName(params);
-            print(getTopics().getPartitionedStats(topic, perPartition, getPreciseBacklog));
+            print(topics.getPartitionedStats(topic, perPartition, getPreciseBacklog, subscriptionBacklogSize));


@MarvinCai it seems that getTopics() was changed to topics here unintentionally. Would you mind fixing this?

Fixes #9254 Add ability to fetch backlog size for subscription, add flag in topic-stats partitioned-topic-stats for getting backlog size for subscriptions. Sample output ``` ./pulsar-admin topics partitioned-stats zxc-t/zxc-ns/zxc-p -gpb -sbs { "msgRateIn" : 0.0, "msgThroughputIn" : 0.0, "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesInCounter" : 0, "msgInCounter" : 0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "averageMsgSize" : 0.0, "msgChunkPublished" : false, "storageSize" : 875, "backlogSize" : 585, "publishers" : [ ], "waitingPublishers" : 0, "subscriptions" : { "zxc-sub-1" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "chuckedMessageRate" : 0, "msgBacklog" : 10, "backlogSize" : 585, "msgBacklogNoDelayed" : 10, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "unackedMessages" : 0, "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 0, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "consumersAfterMarkDeletePosition" : { }, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0 }, "zxc-sub-2" : { "msgRateOut" : 0.0, "msgThroughputOut" : 0.0, "bytesOutCounter" : 0, "msgOutCounter" : 0, "msgRateRedeliver" : 0.0, "chuckedMessageRate" : 0, "msgBacklog" : 5, "backlogSize" : 295, "msgBacklogNoDelayed" : 5, "blockedSubscriptionOnUnackedMsgs" : false, "msgDelayed" : 0, "unackedMessages" : 0, "msgRateExpired" : 0.0, "totalMsgExpired" : 0, "lastExpireTimestamp" : 0, "lastConsumedFlowTimestamp" : 0, "lastConsumedTimestamp" : 0, "lastAckedTimestamp" : 0, "lastMarkDeleteAdvancedTimestamp" : 0, "consumers" : [ ], "isDurable" : true, "isReplicated" : false, "consumersAfterMarkDeletePosition" : { }, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0 } }, "replication" : { }, "nonContiguousDeletedMessagesRanges" : 0, "nonContiguousDeletedMessagesRangesSerializedSize" : 0, "metadata" : { "partitions" : 5 }, "partitions" : { } } ``` In ManagedLedgerImpl add API to get backlog size starting from specific position. In Admin Rest API and CLI add option to get subscription backlog. This change added tests and can be verified as follows: Added check in existing test to verify backlog size. (cherry picked from commit ab94743)

This version update is convenient for tests in real environment since there's no binary download url for original pulsar `2.8.0-rc-202101252233`. This PR fixes the API incompatibility problems that are introduced by apache/pulsar#9397 and apache/pulsar#9302. Another significant change between these two versions is apache/pulsar#9338, which introduced metadata-store API for cluster resources. This PR fixed the test failure caused by it as well. Since KoP `tests` module only uses one `MockZooKeeper` to manage z-nodes, see `KopProtocolHandlerTestBase#createMockZooKeeper`, the mocked `createConfigurationMetadataStore` method returns `mockedZooKeeper` here instead of a `mockedZooKeeperGlobal` like what Pulsar did in `MockedPulsarServiceBaseTest`. Besides, there's a test bug in `testBrokerHandleTopicMetadataRequest` that was not exposed by the previous Pulsar. This PR fixes it.

MarvinCai added 2 commits January 24, 2021 15:55

Add subscription backlog size info for topicstats.

7f760e7

Fixing code style.

c1f4c35

zymap reviewed Jan 25, 2021

View reviewed changes

zymap assigned MarvinCai Jan 25, 2021

zymap added the component/stats label Jan 25, 2021

zymap added this to the 2.8.0 milestone Jan 25, 2021

sijie added the release/2.7.1 label Jan 25, 2021

Fix unit tests.

5dde05a

MarvinCai requested review from codelipenghui and zymap January 26, 2021 00:00

Fix nameing.

818c611

zymap approved these changes Jan 28, 2021

View reviewed changes

codelipenghui approved these changes Jan 28, 2021

View reviewed changes

sijie approved these changes Jan 30, 2021

View reviewed changes

Merge remote-tracking branch 'apache-pulsar/master' into subscription…

52546bd

…-backlogsize

Merge remote-tracking branch 'apache-pulsar/master' into subscription…

68bc3e8

…-backlogsize

Fix cmd topic npe.

392028b

Merge remote-tracking branch 'apache-pulsar/master' into subscription…

3c85dcc

…-backlogsize

Merge remote-tracking branch 'apache-pulsar/master' into subscription…

9c3978e

…-backlogsize

gaoran10 approved these changes Feb 7, 2021

View reviewed changes

pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java Outdated Show resolved Hide resolved

MarvinCai added 2 commits February 6, 2021 22:16

Fix typo.

7f0cde5

Merge remote-tracking branch 'apache-pulsar/master' into subscription…

9291fca

…-backlogsize

sijie merged commit ab94743 into apache:master Feb 8, 2021

aloyszhang mentioned this pull request Feb 9, 2021

[hotfix] fix problem caused by concurrent merge #9546

Merged

lhotari reviewed Feb 9, 2021

View reviewed changes

codelipenghui added the cherry-picked/branch-2.7 Archived: 2.7 is end of life label Feb 18, 2021

BewareMyPower mentioned this pull request Feb 26, 2021

Bump pulsar to 2.8.0-rc-202102252222 streamnative/kop#386

Merged

This was referenced Dec 24, 2022

[fix][broker] Fix configuration exposeSubscriptionBacklogSizeInPrometheus not working #19049

Open

PIP-235: Add metric for subscription backlog size #19112

Open

sijie mentioned this pull request Dec 30, 2022

ISSUE-19112: PIP-235: Add metric for subscription backlog size streamnative/pulsar-archived#5341

Open

poorbarcode mentioned this pull request Jan 20, 2023

[improve] [admin] Make the default value of param --get-subscription-backlog-size of admin API topics stats true #19302

Merged

4 tasks

	private boolean exposeSubscriptionBacklokSizeInPrometheus = false;
	private boolean exposeSubscriptionBacklogSizeInPrometheus = false;

Add subscription backlog size info for topicstats. #9302

Add subscription backlog size info for topicstats. #9302

Conversation

MarvinCai commented Jan 25, 2021 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarvinCai Jan 25, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarvinCai commented Jan 28, 2021

zymap left a comment

Choose a reason for hiding this comment

MarvinCai commented Jan 28, 2021

MarvinCai commented Jan 28, 2021

MarvinCai commented Jan 29, 2021

sijie commented Jan 30, 2021

MarvinCai commented Jan 30, 2021

MarvinCai commented Jan 30, 2021

MarvinCai commented Jan 30, 2021

MarvinCai commented Jan 30, 2021

MarvinCai commented Jan 31, 2021

MarvinCai commented Jan 31, 2021

MarvinCai commented Jan 31, 2021

MarvinCai commented Feb 1, 2021

MarvinCai commented Feb 1, 2021

MarvinCai commented Feb 1, 2021

MarvinCai commented Feb 1, 2021

MarvinCai commented Feb 1, 2021

MarvinCai commented Feb 1, 2021

zymap commented Feb 2, 2021

MarvinCai commented Feb 2, 2021

MarvinCai commented Feb 2, 2021

MarvinCai commented Feb 3, 2021

MarvinCai commented Feb 3, 2021

zymap commented Feb 4, 2021

MarvinCai commented Feb 6, 2021

gaoran10 left a comment

Choose a reason for hiding this comment

MarvinCai commented Feb 7, 2021

Choose a reason for hiding this comment

MarvinCai commented Jan 25, 2021 •

edited

Loading

MarvinCai Jan 25, 2021 •

edited

Loading