[fix][broker] fix UniformLoadShedder selecet wrong overloadbroker and underloadbroker #19
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
…currently (apache#20971) ### Motivation **Background**: when calling `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>`, Pulsar will read the first entry which is not acknowledged, and respond with the entry write time. The flow is like this: - get the mark deleted position of the subscription - if no backlog, response `-1` - else read the next position of the mark deleted position, and respond with the entry write time. **Issue**: if the command `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` and `consumer.acknowledge` are executed at the same time, the step 2 in above flow will get a position which is larger than the last confirmed position, lead a read entry error. | time | `pulsar-admin topics stats --get-earliest-time-in-backlog <topic name>` | `consumer.acknowledge` | | --- | --- | --- | | 1 | mark deleted position is `3:1` and LAC is `3:2` now | | 2 | the check `whether has backlog` is passed | | 3 | | acknowledged `3:2`, mark deleted position is `3:2` now | | 4 | calculate next position: `3:3` | | 5 | Read `3:3` and get an error: `read entry failed` | Note: the test in PR is not intended to reproduce the issue. ### Modifications Respond `-1` if the next position of the mark deleted position is larger than the LAC
…data. sec ver. (apache#20620) Co-authored-by: wangjinlong <wangjinlong@zhihu.com>
…ageId read reaches lastReadId (apache#20988)
…ic migration (apache#21029) Co-authored-by: Vineeth Polamreddy <vineeth.polamreddy@verizonmedia.com>
### Motivation Removing `webUrl` null-check, because it couldn't be null. ### Modifications Removing `webUrl` null-check
…20782) Co-authored-by: ran <gaoran_10@126.com>
… and not need to expose the private key to the WebSocket Proxy (apache#20923) - [DISCUSS](https://lists.apache.org/thread/jcjwkpcfz4h3fnkhjvtrhf620f91nwqn) - [VOTE](https://lists.apache.org/thread/qn22c97j5pkohvn9yo9hwb29z3r8zx3c)
…ed on topic, when dedup is enabled and no producer is there (apache#20951)
…pache#21035) Motivation: After [PIP-118: reconnect broker when ZooKeeper session expires](apache#13341), the Broker will not shut down after losing the connection of the local metadata store in the default configuration. However, before the ZK client is reconnected, the events of BK online and offline are lost, resulting in incorrect BK info in the memory. You can reproduce the issue by the test `BkEnsemblesChaosTest. testBookieInfoIsCorrectEvenIfLostNotificationDueToZKClientReconnect`(90% probability of reproduce of the issue, run it again if the issue does not occur) Modifications: Refresh BK info in memory after the ZK client is reconnected.
…letion of compaction (apache#21067)
…and schema. (apache#21093) Fixes apache#21075 ### Motivation When the topic is loaded, it will delete the topic-level policy if it is enabled. But if the topic is not loaded, it will directly delete through managed ledger factory. But then we will leave the topic policy there. When the topic is created next time, it will use the old topic policy ### Modifications When deleting the topic, delete the schema and topic policies even if the topic is not loaded.
## Motivation
Handle ack hole case:
For example:
```markdown
Chunk-1 sequence ID: 0, chunk ID: 0, msgID: 1:1
Chunk-2 sequence ID: 0, chunk ID: 1, msgID: 1:2
Chunk-3 sequence ID: 0, chunk ID: 0, msgID: 1:3
Chunk-4 sequence ID: 0, chunk ID: 1, msgID: 1:4
Chunk-5 sequence ID: 0, chunk ID: 2, msgID: 1:5
```
Consumer ack chunk message via ChunkMessageIdImpl that consists of all the chunks in this chunk
message(Chunk-3, Chunk-4, Chunk-5). The Chunk-1 and Chunk-2 are not included in the
ChunkMessageIdImpl, so we should process it here.
## Modification
Ack chunk-1 and chunk-2.
…trategicTwoPhaseCompactor (apache#21091)
…fter RawReader reconnects (apache#21081)
…ing broker deletion event (apache#21083)
### Modifications When upgraded the pulsar version from 2.9.2 to 2.10.3, and the isolated group feature not work anymore. Finally, we found the problem. In IsolatedBookieEnsemblePlacementPolicy, when it gets the bookie rack from the metadata store cache, uses future.isDone() to avoid sync operation. If the future is incomplete, return empty blacklists. The cache may expire due to the caffeine cache `getExpireAfterWriteMillis` config, if the cache expires, the future may be incomplete. (apache#21095 will correct the behavior) In 2.9.2, it uses the sync to get data from the metadata store, we should also keep the behavior.
…eystore TLS and admin API is called (apache#21077)
… the fatal exception (apache#21079) ### Motivation This PIP is to improve the current function exception handler. It will be applied to both Pulsar Function and Pulsar IO Connector. For more details, please refer to `pip-297.md`
Signed-off-by: Jiwe Guo <technoboy@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for running tests for upstream PR apache#21025.