[Bug][broker] Error deserializing message，Invalid unknonwn tag type: 7

[TakaHiR07]

Search before asking

• I searched in the issues and found nothing similar.

Version

1. pulsar-server-version: 2.9.3.0.5 based on 2.9.3
2. client-producer-version: c++ 2.9.3
3. client-consumer-version: go-client

Minimal reproduce step

Continuously send message, receive message and ack.

What did you expect to see?

deserializing message metadata successfully

What did you see instead?

deserializing message metadata error when PersistentMessageExpiryMonitor#expireMessages, which result in topic message can not be cleaned.

markDeletePosition is 48516, so expireMessages need to get the timestamp of 48517. In our production environment, bookie read entry 69134924:48517 successfully, but broker fail in parseMessageMetadata. entry is 3 replication. message retention time is 3 day.

However, the near position such as 48516, 48518, 48519 can read successfully, only 48517 fail.

The stack is as follow

14:12:57.694 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor - [persistent://like/baina/like_user_event_0501016-partition-5][cksinker_like_user_event_0501016] Error deserializing message for expiry check
	java.lang.IllegalArgumentException: Invalid unknonwn tag type: 7
        at org.apache.pulsar.common.api.proto.LightProtoCodec.skipUnknownField(LightProtoCodec.java:270) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.pulsar.common.api.proto.MessageMetadata.parseFrom(MessageMetadata.java:1370) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:435) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:422) ~[org.apache.pulsar-pulsar-common-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.pulsar.client.impl.MessageImpl.getEntryTimestamp(MessageImpl.java:284) ~[org.apache.pulsar-pulsar-client-original-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.pulsar.broker.service.persistent.PersistentMessageExpiryMonitor.lambda$expireMessages$0(PersistentMessageExpiryMonitor.java:77) ~[org.apache.pulsar-pulsar-broker-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.bookkeeper.mledger.impl.OpFindNewest.readEntryComplete(OpFindNewest.java:89) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5]
        at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.lambda$asyncReadEntry0$0(EntryCacheImpl.java:222) ~[org.apache.pulsar-managed-ledger-2.9.3.0.5.jar:2.9.3.0.5]
        at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714) [?:?]
        at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

group stats is as follow

"group_xxx" : {
      "markDeletePosition" : "69134924:48516",
      "readPosition" : "79441322:45498",
      "waitingReadOp" : true,
      "pendingReadOps" : 0,
      "messagesConsumedCounter" : 465871946,
      "cursorLedger" : 79430404,
      "cursorLedgerLastEntry" : 5047,
      "individuallyDeletedMessages" : "[(69134924:48518..69134924:49999],(69136302:-1..69136302:49999]

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[github-actions]

The issue had no activity for 30 days, mark with Stale label.