[Issue 8787][C++] Add reader internal subscription name setter. #8823
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/pulsarbot run-failure-checks |
|
Thanks for the work @RobertIndie , Would you please also help provide the python part of change? |
|
/pulsarbot run-failure-checks |
2 similar comments
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
BewareMyPower
approved these changes
Dec 7, 2020
|
/pulsarbot run-failure-checks |
1 similar comment
|
/pulsarbot run-failure-checks |
|
The test is failed by this. I remember it already fixed in PR #8791. So could you please rebase to the latest master? |
jiazhai
approved these changes
Dec 8, 2020
|
/pulsarbot run-failure-checks |
3 similar comments
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
Motivation Add test for apache#8697 and other shade problems. Modifications splitting pulsar-client-and-admin-shade-test into two parts, before the pulsar-admin and pulsar-client shade tests were together, can cause some potential problems. * Add CI test for shade package * fix style * fix license * separate shade package * fix pom.xml * fix codestyle * remove unused import Co-authored-by: WeijieGuo <reswqa@bupt.edu.cn>
Fixes apache/bookkeeper#2500 ### Motivation The dbStorage_writeCacheMaxSizeMb description is not described clearly in the docs. ### Modifications Update the doc with broker.conf file and sync in different versions.
### Motivation The whole Pulsar website could not be built correctly with some syntax errors. Found the error here https://github.com/apache/pulsar/runs/1483341061?check_suite_focus=true ### Modifications Fix the syntax error.
* update * Update site2/docs/client-libraries-websocket.md Co-authored-by: HuanliMeng <48120384+Huanli-Meng@users.noreply.github.com> Co-authored-by: HuanliMeng <48120384+Huanli-Meng@users.noreply.github.com>
### Motivation license-maven-plugin check fail for ahc.properties, maybe related with license-maven-plugin update in apache#8706  ### Modifications Add ahc.properties to exclude list of license-maven-plugin
--- Fixes apache#8790 *Motivation* The test HealthCheckTest.testBookKeeperDown always failed. That because of we have a change in the `bin/pulsar bookie`. apache#8065 changed the main class name so the bookie process name change to the `MAIN` not `BookieServer`.
…Validation (apache#8771) * [C++] Fixed flaky test: AuthPluginTest.testTlsDetectHttpsWithHostNameValidation * Fixed test assertion
Signed-off-by: xiaolong.ran <rxl@apache.org> Fixes apache#8769 ### Motivation When we use pulsar-admin or pulsarctl to create Go Functions, we need to specify the input topic, but in many cases, we will directly specify the topic name, for example: `input-topic` instead of: `persistent://publlic/default/input-topic`. In the consumer of Go Function, we have the following map structure to map the relationship between topic and consumer, as follows: ``` consumers map[string]pulsar.Consumer ``` When assigning a value to the map structure, we use the value directly passed in from pulsar-admin or pulsarctl as the key of the map, but when using it, we use the full path form (eg:persistent:/ /publlic/default/input-topic), so the corresponding key cannot be found, resulting in the following panic: ``` 2020/12/02 11:38:47.845 [info] Created producer cnx=127.0.0.1:54594 -> 127.0.0.1:6650 topic=persistent://public/default/out-topic producer_name=standalone-0-61 producerID=1 ===============topic name================ topic name: in-topic 11:38:47.846 [pulsar-io-51-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:54594] Subscribing on topic persistent://public/default/in-topic / public/default/go_func 11:38:47.847 [pulsar-io-51-6] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/default/persistent/in-topic-public%2Fdefault%2Fgo_func] Rewind from 13:10 to 13:0 11:38:47.847 [pulsar-io-51-6] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/in-topic] There are no replicated subscriptions on the topic 11:38:47.847 [pulsar-io-51-6] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://public/default/in-topic][public/default/go_func] Created new subscription for 1 11:38:47.847 [pulsar-io-51-6] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:54594] Created subscription on topic persistent://public/default/in-topic / public/default/go_func 2020/12/02 11:38:47.847 asm_amd64.s:1373: [info] Connected consumer subscription=public/default/go_func consumerID=1 name=jrppw topic=persistent://public/default/in-topic 2020/12/02 11:38:47.847 asm_amd64.s:1373: [info] Created consumer name=jrppw topic=persistent://public/default/in-topic subscription=public/default/go_func consumerID=1 2020/12/02 11:38:47.847 log.go:46: [info] Serving InstanceCommunication on port 54013 2020/12/02 11:38:47.848 instance.go:402: [error] the logAppender is nil, if you want to use it, please specify `--log-topic` at startup. 2020/12/02 11:38:47.848 contextFunc.go:30: [info] attempting to discard input ++++++++++++++++++++ ack input msg topic: persistent://public/default/in-topic ++++++++++++++++++++++++ panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x46a1547] goroutine 1 [running]: github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).ackInputMessage(0xc000248000, 0x4a3d420, 0xc00040e000) /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/pf/instance.go:367 +0x1d7 github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).processResult(0xc000248000, 0x4a3d420, 0xc00040e000, 0x0, 0x0, 0x0) /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/pf/instance.go:355 +0x80 github.com/apache/pulsar/pulsar-function-go/pf.(*goInstance).startFunction(0xc000248000, 0x4a23740, 0xc00020e0c0, 0x0, 0x0) /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/pf/instance.go:177 +0x584 github.com/apache/pulsar/pulsar-function-go/pf.Start(0x4818b00, 0x4938eb0) /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/pf/function.go:171 +0x6b main.main() /Users/wolf4j/github.com/apache/pulsar/pulsar-function-go/examples/contextFunc/contextFunc.go:35 +0x39 11:38:47.855 [pulsar-io-51-6] INFO org.apache.pulsar.broker.service.ServerCnx - Closed connection from /127.0.0.1:54594 ``` ### Modifications In this pull request, we define a `TopicName` structure to parse the topic name passed in by `pulsar-admin` or `pulsarctl`. If it is the default, use the full path method for splicing to ensure that the map is used key is the same.
* PIP 68: Exclusive Producer * Added missing enums cases in C++ * Addressed comments * Moved constant to top of file * Fix mistake in previous update * Added handling for topic deletion
Signed-off-by: xiaolong.ran <rxl@apache.org> ### Motivation The changing base on apache#8523 ### Modifications Support key_based batch builder for Go Functions
Master Issue: apache#8676 ### Motivation Support bookkeeper storage for packages management service. ### Modifications - Add bookkeeper storage provider - Add bookkeeper storage unit tests
…e#8764) --- Master Issue: apache#8676 *Motivation* Make the pulsar service has the ability to enable the packages management service. *Modifications* - Add the packages management service in the start up process - Add the related configuration in the configuration file Will add integration tests after the REST API service and Client added.
…thread to avoid deadlock when removing inactive subscriptions. (apache#8820) Fixes apache#8783 Execute removing non-persistent subscription of topic from different thread to avoid deadlock when removing inactive subscriptions. ### Motivation We have non persistent topics with configured removing inactive subscriptions. We've noticed, that removing subscriptions sometimes hangs. After investigation, we've noticed that it hangs on remove operation of ConcurrentOpenHashMap. We've found that removal operation is made inside lambda of forEach method on same collection. Problem was causing not only problems with subscription removal, but after it happened once, problems with creating new subscriptions. ### Modifications We've modified code and run remove operation in other thread to avoid deadlock.
Fixes apache#8787 ### Motivation In the current reader, we set the subscription name combine with a random part. ### Modifications add subscription setter ### Verifying this change ReaderTest#testReaderSubName
Fix: https://github.com/streamnative/pulsar/issues/1804 This PR enable spotbugs for moduel `pulsar-zookeeper`
Main issue: https://github.com/streamnative/pulsar/issues/1768 This PR changes code style related things only
…mers (apache#8837) ### Motivation *The official performance test tool pulsar-perf now supports only one consumer per subscription, not multiple consumers per subscription.* ### Modifications - A new parameter `numSubscriptions` was added, which specifies the number of subscriptions per Topic - Change the definition of `numConsumers`: before: number of consumers per topic; After: the number of consumers per subscription - add new feature: one subscription has more than one consumers - add new parameter: `receiver-queue-size-across-partitions`, which means *Max total size of the receiver queue across partitions*
…aerospike (apache#8842) Fixes https://github.com/streamnative/pulsar/issues/1796, apache#8843 ### Motivation * This is a start, enable spotbugs for the pulsar-io module. ### Modifications * Enable spotbugs for pulsar-io-aws and pulsar-io-aerospike
…che#8824) Fixes apache#8338 ### Motivation >In some scenarios, users use their own function-worker to connect to an existing pulsar cluster. Their own function-worker and pulsar cluster have different authentication methods, In the following code, when both function-worker and client have enabled the authentication and authorization services, the authentication and authorization can take effect. A better way is to separate them. function-worker can enable and disable the authentication service, and the broker-client can also enable and disable the authentication service according to the configuration. ### Modifications Add a configuration called `brokerClientAuthenticationEnabled` in the configuration file, which is disabled by default. It is used to control whether the broker-client of function-worker enable or disable the authentication.
…8816) --- Master Issue: apache#8676 *Motivation* Introduce the pulsar admin operations with packages management service. *Modifications* - Add the admin API for packages management service. This only has client-side code. Will add integration tests when server-side code is merged.
…e retention (apache#8725) When we are trimming the ledgers we are saving the `currentLedger` but as soon as your restart the broker the currentLedger is not containing the lastMessageId (because it is a fresh new ledger). Changes: - add test case on pulsar-broker that reproduces the issue reported but the user - log a message when we are trimming the ledger at lastAddConfirmedEntry - add test case that prevent changes in the future on ManagedLedgerImpl - fix a minor issue in PersistentTopic#getLastMessageId, a return keyword was missing and we continued with a call ti ManagedLedger, the CompletableFuture was already 'completed' so the final result is not changed (but we are saving resources) Fixes apache#8677
…m CheckStyle (apache#8848) * add provided jclouds Signed-off-by: Renkai <gaelookair@gmail.com> * make broker-style gradually conform checkstyle Signed-off-by: Renkai <gaelookair@gmail.com> * make broker-style gradually conform checkstyle Signed-off-by: Renkai <gaelookair@gmail.com> * make broker-style gradually conform checkstyle Signed-off-by: Renkai <gaelookair@gmail.com> * make broker-style gradually conform checkstyle Signed-off-by: Renkai <gaelookair@gmail.com>
Signed-off-by: xiaolong.ran <rxl@apache.org> ### Motivation * Enable spotbugs for the pulsar-client-tools-test module. ### Modifications * Enable spotbugs for the pulsar-client-tools-test module. ### Verifying this change ``` mvn clean install -pl pulsar-client-tools-test ```
### Motivation When the value of "/bookies" in ZooKeeper is changed, Brokers are notified of the change. After the notification to Brokers, only one of the two BookKeeper clients that Brokers have seems to reflect the change. **Two BookKeeper clients that Brokers have** - https://github.com/apache/pulsar/blob/ac0c6e41f0ebe3c900bb31e41c8d40b3f60b19df/pulsar-broker/src/main/java/org/apache/pulsar/broker/ManagedLedgerClientFactory.java#L81 - https://github.com/apache/pulsar/blob/102fa9de03509b86e47f58ab8e1c0dde2095da3b/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/BookkeeperSchemaStorage.java#L105 The client of `BookkeeperSchemaStorage` don't seems to reflect the change. The cause is that `ZkBookieRackAffinityMapping#onUpdate` don't run. I confirmed that I change ZkBookieRackAffinityMapping instances to use same `ZooKeeperDataCache` and `ZkBookieRackAffinityMapping#onUpdate` works. ### Modification - Set the argument of `ZkBookieRackAffinityMapping#setConf` to `bookieMappingCache` - Move `updateRacksWithHost()`
Add provided `jclouds-blobstore` for module `tiered-storage`. A provided dependency will not change the build result of a maven project, but it will provide information to make IDEs easier to parse and analyze the project dependencies rather than a highly customized shaded dependency.
sijie
approved these changes
Dec 8, 2020
sijie
added
component/c++
area/client
type/feature
zymap
pushed a commit
to zymap/pulsar
that referenced
this pull request
Dec 9, 2020
…he#8823) Master Issue: apache#8787 ### Motivation Currently, the reader subscription name can only be generated internally randomly in the C++ client. Java client part is at apache#8801 ### Modifications Add a setter for the reader's internal subscription name. ### Verifying this change This change is already covered by existing tests, such as *testSubscriptionNameSetting*, *testSetSubscriptionNameAndPrefix* and *testMultiSameSubscriptionNameReaderShouldFail*.
codelipenghui
pushed a commit
that referenced
this pull request
Dec 21, 2020
Master Issue: #8787 ### Motivation Currently, the reader subscription name can only be generated internally randomly in the C++ client. Java client part is at #8801 ### Modifications Add a setter for the reader's internal subscription name. ### Verifying this change This change is already covered by existing tests, such as *testSubscriptionNameSetting*, *testSetSubscriptionNameAndPrefix* and *testMultiSameSubscriptionNameReaderShouldFail*. (cherry picked from commit 408f9e6)
|
depends by #8905 |
codelipenghui
pushed a commit
that referenced
this pull request
Dec 21, 2020
Master Issue: #8787 ### Motivation Currently, the reader subscription name can only be generated internally randomly in the C++ client. Java client part is at #8801 ### Modifications Add a setter for the reader's internal subscription name. ### Verifying this change This change is already covered by existing tests, such as *testSubscriptionNameSetting*, *testSetSubscriptionNameAndPrefix* and *testMultiSameSubscriptionNameReaderShouldFail*. (cherry picked from commit 408f9e6)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master Issue: #8787
Motivation
Currently, the reader subscription name can only be generated internally randomly in the C++ client.
Java client part is at #8801
Modifications
Add a setter for the reader's internal subscription name.
Verifying this change
This change is already covered by existing tests, such as testSubscriptionNameSetting, testSetSubscriptionNameAndPrefix and testMultiSameSubscriptionNameReaderShouldFail.