Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix returned wrong hash ranges for the consumer with same consumer name #12212

Conversation

codelipenghui
Copy link
Contributor

Currently, we are using the consumer name to generate the hash ranges to the admin client.
If there are consumers with the same name, we will get same hash ranges for different consumers,
this will confuse when troubleshooting issue. The following is an example:

"consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 46320,
        "msgOutCounter" : 1020,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "5253f",
        "availablePermits" : -20,
        "unackedMessages" : 1000,
        "avgMessagesPerEntry" : 56,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:11494",
        "lastAckedTimestamp" : 1632731049993,
        "lastConsumedTimestamp" : 1632731030268,
        "keyHashRanges" : [ "[0, 16384]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54702",
        "connectedSince" : "2021-09-27T16:23:49.891+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19505",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54708",
        "connectedSince" : "2021-09-27T16:23:59.031+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19514",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54717",
        "connectedSince" : "2021-09-27T16:24:03.927+08:00",
        "clientVersion" : "2.8.1"
      } ],

The fix is to use the equals method of the consumer to generate the key hash ranges.
New tests added.

Currently, we are using the consumer name to generate the hash ranges to the admin client.
If there are consumers with the same name, we will get same hash ranges for different consumers,
this will confuse when troubleshooting issue. The following is an example:

```
"consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 46320,
        "msgOutCounter" : 1020,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "5253f",
        "availablePermits" : -20,
        "unackedMessages" : 1000,
        "avgMessagesPerEntry" : 56,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:11494",
        "lastAckedTimestamp" : 1632731049993,
        "lastConsumedTimestamp" : 1632731030268,
        "keyHashRanges" : [ "[0, 16384]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54702",
        "connectedSince" : "2021-09-27T16:23:49.891+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19505",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54708",
        "connectedSince" : "2021-09-27T16:23:59.031+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19514",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54717",
        "connectedSince" : "2021-09-27T16:24:03.927+08:00",
        "clientVersion" : "2.8.1"
      } ],
```

The fix is to use the equals method of the consumer to generate the key hash ranges.
New tests added.
@codelipenghui codelipenghui self-assigned this Sep 27, 2021
@codelipenghui codelipenghui added this to the 2.9.0 milestone Sep 27, 2021
@codelipenghui codelipenghui added release/2.8.2 component/key-shared doc-not-needed Your PR changes do not impact docs type/bug The PR fixed a bug or issue reported a bug labels Sep 27, 2021
Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui merged commit 9abd6d3 into apache:master Sep 28, 2021
@codelipenghui codelipenghui deleted the penghui/fix-hash-ranges-with-same-consumer-name branch September 28, 2021 11:46
codelipenghui added a commit that referenced this pull request Sep 29, 2021
…me (#12212)

Currently, we are using the consumer name to generate the hash ranges to the admin client.
If there are consumers with the same name, we will get same hash ranges for different consumers,
this will confuse when troubleshooting issue. The following is an example:

```
"consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 46320,
        "msgOutCounter" : 1020,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "5253f",
        "availablePermits" : -20,
        "unackedMessages" : 1000,
        "avgMessagesPerEntry" : 56,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:11494",
        "lastAckedTimestamp" : 1632731049993,
        "lastConsumedTimestamp" : 1632731030268,
        "keyHashRanges" : [ "[0, 16384]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54702",
        "connectedSince" : "2021-09-27T16:23:49.891+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19505",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54708",
        "connectedSince" : "2021-09-27T16:23:59.031+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19514",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54717",
        "connectedSince" : "2021-09-27T16:24:03.927+08:00",
        "clientVersion" : "2.8.1"
      } ],
```

The fix is to use the equals method of the consumer to generate the key hash ranges.
New tests added.

(cherry picked from commit 9abd6d3)
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Sep 29, 2021
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Feb 2, 2022
…me (apache#12212)

Currently, we are using the consumer name to generate the hash ranges to the admin client.
If there are consumers with the same name, we will get same hash ranges for different consumers,
this will confuse when troubleshooting issue. The following is an example:

```
"consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 46320,
        "msgOutCounter" : 1020,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "5253f",
        "availablePermits" : -20,
        "unackedMessages" : 1000,
        "avgMessagesPerEntry" : 56,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:11494",
        "lastAckedTimestamp" : 1632731049993,
        "lastConsumedTimestamp" : 1632731030268,
        "keyHashRanges" : [ "[0, 16384]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54702",
        "connectedSince" : "2021-09-27T16:23:49.891+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19505",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54708",
        "connectedSince" : "2021-09-27T16:23:59.031+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19514",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54717",
        "connectedSince" : "2021-09-27T16:24:03.927+08:00",
        "clientVersion" : "2.8.1"
      } ],
```

The fix is to use the equals method of the consumer to generate the key hash ranges.
New tests added.

(cherry picked from commit 9abd6d3)
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
…me (apache#12212)

Currently, we are using the consumer name to generate the hash ranges to the admin client.
If there are consumers with the same name, we will get same hash ranges for different consumers,
this will confuse when troubleshooting issue. The following is an example:

```
"consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 46320,
        "msgOutCounter" : 1020,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "5253f",
        "availablePermits" : -20,
        "unackedMessages" : 1000,
        "avgMessagesPerEntry" : 56,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:11494",
        "lastAckedTimestamp" : 1632731049993,
        "lastConsumedTimestamp" : 1632731030268,
        "keyHashRanges" : [ "[0, 16384]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54702",
        "connectedSince" : "2021-09-27T16:23:49.891+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19505",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54708",
        "connectedSince" : "2021-09-27T16:23:59.031+08:00",
        "clientVersion" : "2.8.1"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chunkedMessageRate" : 0.0,
        "consumerName" : "my-name",
        "availablePermits" : 10,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "10:19514",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRanges" : [ "[16385, 40960]", "[40961, 65536]" ],
        "metadata" : { },
        "address" : "/127.0.0.1:54717",
        "connectedSince" : "2021-09-27T16:24:03.927+08:00",
        "clientVersion" : "2.8.1"
      } ],
```

The fix is to use the equals method of the consumer to generate the key hash ranges.
New tests added.
@codelipenghui codelipenghui restored the penghui/fix-hash-ranges-with-same-consumer-name branch May 17, 2022 01:21
@codelipenghui codelipenghui deleted the penghui/fix-hash-ranges-with-same-consumer-name branch May 17, 2022 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-picked/branch-2.8 Archived: 2.8 is end of life doc-not-needed Your PR changes do not impact docs release/2.8.2 release/2.9.0 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants