[Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331

nlu90 · 2020-02-15T00:54:30Z

Motivation

Adding a new command get-message-by-id to the pulsar-admin which allows user to look at a single message by providing ledger id and entry id.

Modifications

pulsar-admin includes the new command get-message-by-id
pulsar-broker v1/v2 apis to handle the get-message-by-id request from pulsar-admin
managedCursor to read from ledgers with given ledger id and entry id

Verifying this change

Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): no
The public API: no
The schema: no
The default values of configurations: no
The wire protocol: no
The rest endpoints: yes
The admin cli options: yes
Anything that affects deployment: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

nlu90 · 2020-02-15T01:03:40Z

Test result on my local laptop:

$ ./bin/pulsar-admin topics get-message-by-id persistent://public/default/my-topic -s "first-subscription" -l 10 -e 0
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 68 65 6c 6c 6f 2d 70 75 6c 73 61 72             |hello-pulsar    |
+--------+-------------------------------------------------+----------------+```

codelipenghui · 2020-02-15T03:01:26Z

@nlu90 I noticed that to get the message content by ID needs to specify a subscription name. Sorry I didn't clarify this matter in the issue.

Is it better to not require the user to specify the message ID? No matter which subscription name is specified, the returned message content is the same. It's different with peek messages, users can peek one or more messages by peeking messages. So we don’t need to consider whether the message is acknowledged or not.

Future more, the entry may represent a batch message. If I understand correctly, if it is a batch message, the format of the information received by the user may not be easy to read. And the message can encoded by different schema, we need to leverage by pulsar schema. This part can also be solved by a separate PR. At present, the peek message is not handled.

nlu90 · 2020-02-15T06:43:25Z

@codelipenghui

For your first suggestion about removing subscription, I actually considered it when I was first implementing this functionality. But later I kept it there due to implementation simplicity. Now I removed it and will create a one-time subscription when fetching the message internally. Please take a look and let me know if you have any other suggestions.

For your second suggestion -- handling batch message, I thins we can handle it it in a separate issue and PR.

nlu90 · 2020-02-15T06:44:25Z

Now the command works as following:

$ ./bin/pulsar-admin topics get-message-by-id persistent://public/default/my-topic -l 10 -e 0
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 68 65 6c 6c 6f 2d 70 75 6c 73 61 72             |hello-pulsar    |
+--------+-------------------------------------------------+----------------+```

codelipenghui

I left a comment related to using ManagedLedger to get a message directly. And please also update the documentation for the new admin method.

codelipenghui · 2020-02-15T07:12:24Z

managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedCursor.java

+     * @param callback
+     * @param ctx
+     */
+    void asyncGetMessageById(long ledgerId, long entryId, IndividualDeletedEntries deletedEntries,


According to what we discussed before, We don't need to specify a subscription name for getting a message by ID. So I think we can get a message from ManagedLedger directly by https://github.com/apache/pulsar/blob/master/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1494. So that we don't need to create a one-time subscription.

codelipenghui · 2020-02-15T07:13:22Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

+        Entry entry = null;
+        try {
+            PersistentSubscription subscription =
+                    (PersistentSubscription) topic.createSubscription(subName, InitialPosition.Earliest, false).get();


Same as comments above

codelipenghui · 2020-02-15T07:16:22Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Subscription.java

@@ -79,6 +80,8 @@ default long getNumberOfEntriesDelayed() {

    CompletableFuture<Entry> peekNthMessage(int messagePosition);

+    CompletableFuture<Entry> getMessageById(long ledgerId, long entryId);


If we can use ManagedLedger directly, this method also can be removed.

Good suggestion. Fixed them in the following update.

codelipenghui · 2020-02-16T13:10:21Z

@nlu90 Thanks for achieving it. The change looks good to me.
I think the asyncGetMessageById method should remove from ManagedCursor.java right?

nlu90 · 2020-02-17T05:28:45Z

@codelipenghui Thanks for all the suggestions. I removed the asyncGetMessageById .

@sijie I think this feature should not be labelled with component/functions.

sijie · 2020-02-17T06:59:09Z

@nlu90 my bad. fixed it.

sijie · 2020-02-17T07:03:48Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

+                }
+            }, null);
+
+            entry = future.get(1000, TimeUnit.MILLISECONDS);


Sorry for being late in reviewing this pull request. I would suggest implementing this using AsyncResponse. We have tried to move away from using sync methods.

You can check internalCreateSubscription on how to use AsyncResponse.

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

Line 1305 in df15210

protected void internalCreateSubscription(AsyncResponse asyncResponse, String subscriptionName,

codelipenghui · 2020-02-17T07:24:56Z

@sijie @nlu90 I added doc-required tag since this is a new feature for pulsar-admin. It's better to add a new paragraph in https://github.com/apache/pulsar/blob/master/site2/docs/reference-pulsar-admin.md.

nlu90 · 2020-02-18T01:33:25Z

@sijie updated with the asyncResponse to construct the result

@codelipenghui added the doc into the following two md file: reference-pulsar-admin.md and admin-api-persistent-topics.md

sijie

@nlu90 overall looks good. one small comment regarding releasing the entry.

sijie · 2020-02-18T02:24:21Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

+
+        PersistentTopic topic = (PersistentTopic) getTopicReference(topicName);
+        ManagedLedgerImpl ledger = (ManagedLedgerImpl) topic.getManagedLedger();
+        Entry entry = null;


I don't think this is needed any more.

sijie · 2020-02-18T02:27:16Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

+                    clientAppId(), ledgerId, entryId, topicName, exception);
+            asyncResponse.resume(new RestException(exception));
+        } finally {
+            if (entry != null) {


move this finally block to line 1520

…mnative/pulsar into neng/add-get-message-by-id

sijie · 2020-02-18T07:02:50Z

@nlu90

It seems the code can not be compiled.

[ERROR] /home/runner/work/pulsar/pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:[1540,24] method internalPeekNthMessage(java.lang.String,int,boolean) is already defined in class org.apache.pulsar.broker.admin.impl.PersistentTopicsBase

sijie · 2020-02-18T07:04:39Z

pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java

+                @Override
+                public void readEntryComplete(Entry entry, Object ctx) {
+                    try {
+                        asyncResponse.resume(generateResponseWithEntry(entry));


I think we need to release the Entry it after using it.

nlu90 · 2020-02-18T17:38:56Z

@sijie The problem is caused by some conflicting commit. I've fixed them now. Also added the entry.release() logic to the proper place.

sijie

@nlu90 awesome contribution!

…e-by-id

sijie · 2020-04-21T20:16:24Z

/pulsarbot run-failure-checks

Anonymitaet · 2020-06-10T08:22:48Z

Docs have been added:

apache#6331) Fixes apache#2912 ### Motivation Adding a new command `get-message-by-id` to the pulsar-admin which allows user to look at a single message by providing ledger id and entry id. ### Modifications - pulsar-admin includes the new command `get-message-by-id` - pulsar-broker v1/v2 apis to handle the get-message-by-id request from pulsar-admin - managedCursor to read from ledgers with given ledger id and entry id

add get-message-by-id cmd into pulsar-admin

25df99e

nlu90 requested review from sijie and codelipenghui and removed request for sijie February 15, 2020 00:54

remove subscription name parameter from request

532a76d

codelipenghui reviewed Feb 15, 2020

View reviewed changes

use ledger to get entry

feced50

sijie assigned nlu90 Feb 16, 2020

sijie added the area/function label Feb 16, 2020

sijie added this to the 2.6.0 milestone Feb 16, 2020

remove asyncGetMessageById

a7ff070

sijie added area/admin and removed area/function labels Feb 17, 2020

sijie reviewed Feb 17, 2020

View reviewed changes

codelipenghui added the doc-required Your PR changes impact docs and you will update later. label Feb 17, 2020

nlu90 added 2 commits February 17, 2020 17:24

use asyncResponse to generate response and add doc

ec12604

add doc to reference-pulsar-admin

7f6d4b1

Merge branch 'master' into neng/add-get-message-by-id

5ebd9c9

sijie reviewed Feb 18, 2020

View reviewed changes

clean up method internalGetMessageById

9d1c1e6

Merge branch 'neng/add-get-message-by-id' of https://github.com/strea…

882db05

…mnative/pulsar into neng/add-get-message-by-id

sijie reviewed Feb 18, 2020

View reviewed changes

resolve conflicts

b475e8f

Merge branch 'master' into neng/add-get-message-by-id

f772687

sijie approved these changes Apr 2, 2020

View reviewed changes

Merge remote-tracking branch 'apache/master' into neng/add-get-messag…

b7e89a3

…e-by-id

codelipenghui approved these changes Apr 22, 2020

View reviewed changes

codelipenghui merged commit d3ac61f into apache:master Apr 22, 2020

sijie mentioned this pull request May 20, 2020

[discussion] Pulsar release 2.6.0 #5819

Closed

Anonymitaet removed the doc-required Your PR changes impact docs and you will update later. label Jun 10, 2020

Anonymitaet deleted the neng/add-get-message-by-id branch June 10, 2020 08:23

Technoboy- mentioned this pull request Jan 21, 2022

[Broker] Fix call sync method in async rest api for internalGetMessageById. #13876

Merged

1 task

BewareMyPower mentioned this pull request May 18, 2023

[Bug] Topic "peek" (admin/v2/persistent/{tenant}/{namespace}/{topic}/ledger/{ledgerId}/entry/{entryId}) management API returns undocumented and inconsistent data when batch messages are accessed #20258

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331

[Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331

nlu90 commented Feb 15, 2020 •

edited

Loading

nlu90 commented Feb 15, 2020

codelipenghui commented Feb 15, 2020

nlu90 commented Feb 15, 2020

nlu90 commented Feb 15, 2020

codelipenghui left a comment

codelipenghui Feb 15, 2020

codelipenghui Feb 15, 2020

codelipenghui Feb 15, 2020

nlu90 Feb 15, 2020

codelipenghui commented Feb 16, 2020

nlu90 commented Feb 17, 2020

sijie commented Feb 17, 2020

sijie Feb 17, 2020

codelipenghui commented Feb 17, 2020

nlu90 commented Feb 18, 2020

sijie left a comment

sijie Feb 18, 2020

sijie Feb 18, 2020

sijie commented Feb 18, 2020

sijie Feb 18, 2020

nlu90 commented Feb 18, 2020

sijie left a comment

sijie commented Apr 21, 2020

Anonymitaet commented Jun 10, 2020

		@@ -79,6 +80,8 @@ default long getNumberOfEntriesDelayed() {

		CompletableFuture<Entry> peekNthMessage(int messagePosition);

		CompletableFuture<Entry> getMessageById(long ledgerId, long entryId);

[Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331

[Issue 2912][pulsar-admin] add get-message-by-id cmd into pulsar-admin #6331

Conversation

nlu90 commented Feb 15, 2020 • edited Loading

Motivation

Modifications

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

nlu90 commented Feb 15, 2020

codelipenghui commented Feb 15, 2020

nlu90 commented Feb 15, 2020

nlu90 commented Feb 15, 2020

codelipenghui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codelipenghui commented Feb 16, 2020

nlu90 commented Feb 17, 2020

sijie commented Feb 17, 2020

Choose a reason for hiding this comment

codelipenghui commented Feb 17, 2020

nlu90 commented Feb 18, 2020

sijie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sijie commented Feb 18, 2020

Choose a reason for hiding this comment

nlu90 commented Feb 18, 2020

sijie left a comment

Choose a reason for hiding this comment

sijie commented Apr 21, 2020

Anonymitaet commented Jun 10, 2020

nlu90 commented Feb 15, 2020 •

edited

Loading