Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[admin] Enable peeking encrypted batch messages #11244

Merged
merged 1 commit into from
Jul 7, 2021

Conversation

massakam
Copy link
Contributor

@massakam massakam commented Jul 7, 2021

Motivation

We can peek non-batched messages successfully, even if they are encrypted:

$ ./bin/pulsar-admin topics peek-messages -s sub1 persistent://public/default/non-batch

Message ID: 2:0
Tenants:
"publish-time    2021-07-07T13:43:07.621+09:00"
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| f5 ac 6c 84 87 21 ec 67 8c 84 62 1c d5 ff 1f e0 |..l..!.g..b.....|
|00000010| 6a 87 be 62 10                                  |j..b.           |
+--------+-------------------------------------------------+----------------+

On the other hand, if batched messages are encrypted, an exception will be thrown when peeking:

$ ./bin/pulsar-admin topics peek-messages -s sub1 persistent://public/default/batch

13:48:29.578 [AsyncHttpClient-7-1] ERROR org.apache.pulsar.client.admin.internal.TopicsImpl - Exception occurred while trying to get BatchMsgId: 0:0:0
java.lang.IllegalArgumentException: Invalid unknonwn tag type: 6
        at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:270) ~[pulsar-common.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.common.api.proto.SingleMessageMetadata.parseFrom(SingleMessageMetadata.java:470) ~[pulsar-common.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.common.protocol.Commands.deSerializeSingleMessageInBatch(Commands.java:1624) ~[pulsar-common.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.client.admin.internal.TopicsImpl.getIndividualMsgsFromBatch(TopicsImpl.java:1558) ~[pulsar-client-admin-original.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.client.admin.internal.TopicsImpl.getMessagesFromHttpResponse(TopicsImpl.java:1540) ~[pulsar-client-admin-original.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.client.admin.internal.TopicsImpl.access$100(TopicsImpl.java:90) ~[pulsar-client-admin-original.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.client.admin.internal.TopicsImpl$15.completed(TopicsImpl.java:1027) ~[pulsar-client-admin-original.jar:2.9.0-SNAPSHOT]
        at org.apache.pulsar.client.admin.internal.TopicsImpl$15.completed(TopicsImpl.java:1022) ~[pulsar-client-admin-original.jar:2.9.0-SNAPSHOT]
        ...

This is probably because the payload of the batch message is encrypted and it fails to retrieve the individual payloads from it.

Modifications

When an encrypted message is peeked, the broker returns a response with the value of the X-Pulsar-Is-Encrypted header set to true. Then, admin client treats it as a non-batch message and creates a MessageImpl instance. This fix enables us to peek the message without throwing an exception.

$ ./bin/pulsar-admin topics peek-messages -s sub1 persistent://public/default/batch

Message ID: 17:0
Tenants:
"X-Pulsar-batch-size    -292"
"X-Pulsar-num-batch-message    1"
"publish-time    2021-07-07T16:10:45.014+09:00"
         +-------------------------------------------------+
         |  0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f |
+--------+-------------------------------------------------+----------------+
|00000000| 73 3c bf c9 d9 c7 bd 56 6f ac 71 e8 8d 7b fa 33 |s<.....Vo.q..{.3|
|00000010| 2a 46 1e 9e 30 bb e2 7e ff 8c 41 9e a2          |*F..0..~..A..   |
+--------+-------------------------------------------------+----------------+

@massakam massakam added this to the 2.9.0 milestone Jul 7, 2021
@massakam massakam self-assigned this Jul 7, 2021
@merlimat merlimat merged commit d01ecb0 into apache:master Jul 7, 2021
@massakam massakam deleted the peek-encrypted-messages branch July 8, 2021 01:06
codelipenghui pushed a commit that referenced this pull request Jul 30, 2021
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants