Closed
Conversation
…es to DLQ (apache#17317) (cherry picked from commit 0909853)
…che#16365) * Improve the package download process --- *Motivation* Improve the package download process to handle the download body more efficient. (cherry picked from commit 1fe8c06)
…LQ (apache#17326) (cherry picked from commit 2c1d213)
…che#15814) (cherry picked from commit d87cab2)
…17257) ### Motivation `ManagedLedgerMBeanImpl` will execute method `refreshStats` immediately after it is created, but if `refreshStats` is executed too fast, will throws `IllegalArgumentException`. - `refreshStats` execute immediately https://github.com/apache/pulsar/blob/fd9489771959f3e722656e4b70d4bd891a13f690/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryImpl.java#L198-L199 - throw `IllegalArgumentException` if executed too fast https://github.com/apache/pulsar/blob/1de80e0684ec5c13b6edcd217af62d74d8677f04/pulsar-common/src/main/java/org/apache/pulsar/common/stats/Rate.java#L62-L64 `ManagedLedgerMBeanImpl.refreshStats` will immediatelly execute after create ### Modifications Skip refreshing stats, just like `ManagedLedgerFactoryMBeanImpl.refreshStats` https://github.com/apache/pulsar/blob/1de80e0684ec5c13b6edcd217af62d74d8677f04/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerFactoryMBeanImpl.java#L38-L44 (cherry picked from commit 1d1f75e)
- GitHub Actions build queue is long - reduce overhead of cherry-picking by cancelling previous jobs
(cherry picked from commit 2018595)
### Motivation - prometheus `simpleclient` has broken api, it makes `pulsar-client-origin` can not well used with other libraries - specify `OpenMetrics` This PR contains metric name broken changes. ### Modifications Bump prometheus client version from 0.5.0 to 0.15.0 ### changes - bump the client version - rename the field prometheus client `UNTYPED` to `UNKNOWN` - `OpenMetrics`'s counter name needs a `_total` suffix ### metrics name broken changes - rename `pulsar_expired_token_count` to `pulsar_expired_token_total` - rename `pulsar_authentication_success_count` to `pulsar_authentication_success_total` - rename `pulsar_authentication_failures_count` to `pulsar_authentication_failures_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_source_received_total_1min` to `pulsar_source_received_1min_total` - rename `pulsar_source_written_total_1min` to `pulsar_source_written_1min_total` - rename `pulsar_source_source_total_1min` to `pulsar_source_source_exceptions_1min_total ` - rename `pulsar_source_system_exceptions_total_1min` to `pulsar_source_system_exceptions_1min_total ` - rename `pulsar_function_received_total_1min` to `pulsar_function_received_1min_total ` - rename `pulsar_function_user_exceptions_total_1min` to `pulsar_function_user_exceptions_1min_total` - rename `pulsar_function_system_exceptions_total_1min` to `pulsar_function_system_exceptions_1min_total` - rename `pulsar_function_processed_successfully_total_1min` to `pulsar_function_processed_successfully_1min_total` - rename `pulsar_function_received_total_1min` to `pulsar_function_received_1min_total` - rename `pulsar_function_user_exceptions_total_1min` to `pulsar_function_user_exceptions_1min_total` - rename `pulsar_function_system_exceptions_total_1min` to `pulsar_function_system_exceptions_1min_total` - rename `pulsar_function_processed_successfully_total_1min` to `pulsar_function_processed_successfully_1min_total` - rename `pulsar_sink_received_total_1min` to `pulsar_sink_received_1min_total` - rename `pulsar_sink_written_total_1min` to `pulsar_sink_written_1min_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_sink_system_exceptions_total_1min` to `pulsar_sink_system_exceptions_1min_total` - rename `pulsar_sink_received_total_1min` to `pulsar_sink_received_1min_total` - rename `pulsar_sink_written_total_1min` to `pulsar_sink_written_1min_total` - rename `pulsar_sink_sink_exceptions_total_1min` to `pulsar_sink_sink_exceptions_1min_total` - rename `pulsar_sink_system_exceptions_total_1min` to `pulsar_sink_system_exceptions_1min_total` ### Documentation Check the box below or label this PR directly. Need to update docs? - [x] `doc-required` As mentioned above, the metrics name has changed (cherry picked from commit c0b4d57)
### Motivation prometheus client 0.16.0 contains some approvements that we can benefit from. Thanks for @dave2wave @michaeljmarshall the reminder and pointing out. > [ENHANCEMENT] Reduce the number of core threads in HTTPServer from 5 to 1. The HTTPServer will still start up to 5 threads on demand if there are parallel requests, but it will use only 1 thread as long as requests are sequential (prometheus/client_java#786). [ENHANCEMENT] Optimize metric name sanitization: Replace the regular expression with a hard-coded optimized algorithm to improve performance (prometheus/client_java#777). Thanks @fwbrasil See https://github.com/prometheus/client_java/releases ### Modifications Bump prometheus client version from 0.15.0 to 0.16.0 ### Documentation Check the box below or label this PR directly. Need to update docs? - [x] `doc-not-needed` dependency updates, no need doc (cherry picked from commit 948000b)
…tations (apache#16785) Currently, `FieldDoc` annotation has `required` and `defaultValue` config. But `IOConfigUtils` didn't deal with those. The connector we implemented deserialized the configuration using `IOConfigUtils.loadWithSecrets()` - IOConfigUtils support required and defaultValue annotations. (cherry picked from commit 7694803)
…figuration (apache#16408) (cherry picked from commit e4dcf5a)
* [Fix][Flaky-test] Fix testConsumeTxnMessage Master apache#14109 ## Motivation The transaction commit is async, so the consumer can still receive message when the consumer rebuilds. ## Modification Add Awaitility.await() for check-ing whether the ongoingTxns = 0. (cherry picked from commit c29503e)
…d computation (apache#16937) (cherry picked from commit eb833e6)
…16983) --- *Motivation* We found there has a block between the pulsar-ordered executor and the pulsar-stats-updater executor. The pulsar-ordered executor is trying to createManagedLedgerOffloader, and the pulsar-stats-updater is getting the compactor. Both them want to get the lock. We have an improvement about the `createManagedLedgerOffloader` before. apache#15883 We are using `getCompactor(false)` for the stats related operations. The `getCompactor` is guarded by `synchronized`. Actually, the stats just want to get the current compactor without initializing it. We don't need to use `synchronized` to guard this operation. *Modification* Remove unnecessary `synchronized` on the `getCompactor` method. (cherry picked from commit 4d5ecba)
) (cherry picked from commit 5262e6c)
Co-authored-by: congbobo184 <congbobo184@github.com> (cherry picked from commit 176b0d6)
…ssageContainer (apache#17019) (cherry picked from commit 21dc668)
* [fix][tiered-storage] move the state check forward --- *Motivation* Move the close check forward to avoid `getLastAddConfirmed()` get an NPE. If the state is closed. That means the resource is closed and the `OffloadIndexBlock` has been recycled. Which will cause an NPE when `getLastAddCOnfirmed()`. (cherry picked from commit ee0ea3a)
(cherry picked from commit 4b757cf)
…essages have been written to the TCP/IP connection (apache#17043) (cherry picked from commit 2bc933e)
…pache#17045) (cherry picked from commit e0ff3d7)
(cherry picked from commit 2c2b75e)
(cherry picked from commit 16adb61)
…sgs (apache#17093) (cherry picked from commit 4b98b23)
(cherry picked from commit 882fcfb)
…) in load computation (apache#16937)" This reverts commit 688fb76.
(cherry picked from commit 346b04a)
…ache#18906) ### Motivation Cherry-pick apache#17503 to release 2.10.3 and run tests. ### Modifications Cherry-pick apache#17503 to release 2.10.3.
…ache#17022) Master Issue: apache#16912 - [x] `doc-not-needed` cc @shoothzj @eolivelli @nicoloboschi @Technoboy- Signed-off-by: tison <wander4096@gmail.com> (cherry picked from commit 7747795)
…ster specified (apache#17704) (apache#18919) Co-authored-by: fengyubiao <yubiao.feng@streamnative.io>
…nt for C++-owned objects (apache#16535) (apache#18921) Co-authored-by: Zac Bentley <zbentley@users.noreply.github.com>
(cherry picked from commit a951549)
apache#18943) when `MLPendingAckStoreProvider` init PendingAckStore gets the ManagedLedger config throw exception, we don't handle the exception. and the `pendingAckStoreFeture` can't be complete, topic unload will use this future to close the pendingAck. https://github.com/apache/pulsar/blob/3011946a5c3b64ed7c08b6bfb1f6492f8aaaca9c/pulsar-broker/src/main/java/org/apache/pulsar/broker/transaction/pendingack/impl/MLPendingAckStoreProvider.java#L114-L115 when getting managedledger config to fail, `pendingAckStoreFeture` will `completeExceptionally()`; when pendingAckStore init fail, close pendingAckHandle success directly mock get managedLeger config throw exception, then unload can success (cherry picked from commit 1d9956c)
…xt (apache#18971) ### Motivation Since PR apache#18833 can not cherry-pick to `branch-2.10`, create a separate PR. #### Context for Transaction Buffer - If turn on `transactionCoordinatorEnabled`, then `TransactionBuffer` will be initialized when a user topic create. - The `TransactionBuffer` reads the aborted logs of transactions from topic `__transaction_buffer_snapshot` -- this process is called `recovery`. - During recovery, the reading from that snapshot ledger is done via a `Reader`; the reader works like this: ``` while (reader.hasMessageAvailable()){ reader.readNext(); } ``` #### Context for Compaction - After [pip-14](https://github.com/apache/pulsar/wiki/PIP-14:-Topic-compaction), the consumer that enabled feature read-compacted will read messages from the compacted topic instead of the original topic if the task-compaction is done, and read messages from the original topic if task-compaction is not done. - If the data of the last message with key k sent to a topic is null, the compactor will mark all messages for that key as deleted. #### Issue There is a race condition: after executing `reader.hasMessageAvailable`, the following messages have been deleted by compaction-task, so read next will be blocked because there have no messages to read. ---- ### Modifications - If hits this issue, do recover again. ---- #### Why not just let the client try to load the topic again to retry the recover? If the topic load is failed, the client will receive an error response. This is a behavior that we can handle, so should not be perceived by the users.
(cherry picked from commit e07b67f)
…che#18982) ### Motivation delete extra code ### Modifications delete extra code
(cherry picked from commit 22866bd)
…y active cursor (apache#18980) Co-authored-by: LinChen <1572139390@qq.com> Co-authored-by: AloysZhang <lofterzhang@gmail.com> Co-authored-by: Matteo Merli <mmerli@apache.org> Co-authored-by: Michael Marshall <mmarshall@apache.org> Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com>
…rCnx (apache#18987) In the `PulsarDecoder`, we use a single `BaseCommand` object and overwrite it for each incoming protocol message. As a result, it is not safe to publish any references to a proto command to other threads. Here is the single `BaseCommand`: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/PulsarDecoder.java#L99 Here is the method call that resets the object: https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-common/src/main/java/org/apache/pulsar/common/protocol/PulsarDecoder.java#L114 Note that the call to `parseFrom` first calls `clear()`, which resets all values on the object. This PR copies relevant values or objects into other variables. * Replace `command` with `tcId` since the latter is a final variable meant to be published to another thread. * Move logic to copy certain command fields to earlier in method for `handleSubscribe` * Copy `ack` object to new `CommandAck` when there is a broker interceptor. Note that copying this command is likely somewhat costly, so we only do it when there is an interceptor configured. This is a trivial change that is already covered by tests. - [x] `doc-not-needed` This is an internal change. PR in forked repository: michaeljmarshall#8 (cherry picked from commit a408e9e)
…broker interceptors (apache#18997) (cherry picked from commit 1154d0a) (cherry picked from commit b75f068)
efcasado
changed the title
|
@efcasado Please add the following content to your PR description and select a checkbox: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description