Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Message rate out with batches is counting batches/s #466

Closed
merlimat opened this issue Jun 9, 2017 · 3 comments
Closed

Message rate out with batches is counting batches/s #466

merlimat opened this issue Jun 9, 2017 · 3 comments
Assignees
@jai1
Labels
type/bug The PR fixed a bug or issue reported a bug
Milestone
1.18

Comments

@merlimat
Copy link
Contributor

merlimat commented Jun 9, 2017

Expected behavior

When publishing with transparent batching the message rate in is correctly
displayed as msg/s, but the rate out is just counting the batches.

Same thing happens with the backlog count. I think we should have been
displaying an estimated of the messages in the backlog.
@merlimat merlimat added the type/bug The PR fixed a bug or issue reported a bug label Jun 9, 2017
@merlimat merlimat modified the milestone: 1.18 Jun 9, 2017
@jai1
Copy link
Contributor

jai1 commented Jun 9, 2017

Fixed in #462

@jai1 jai1 closed this as completed Jun 9, 2017
@jai1 jai1 self-assigned this Jun 9, 2017
@merlimat
Copy link
Contributor Author

merlimat commented Jun 9, 2017

@jai1 This is unrelated with #462. It happens with Java as well. It's just on the rate out side.

@merlimat merlimat reopened this Jun 9, 2017
@jai1
Copy link
Contributor

jai1 commented Jun 14, 2017

Sorry for the confusion

I think I found the issue
https://github.com/yahoo/pulsar/blob/master/pulsar-broker/src/main/java/com/yahoo/pulsar/broker/service/Consumer.java

Need to change
msgOut.recordEvent(metadataAndPayload.readableBytes());
To
msgOut.recordMultipleEvents(metadataAndPayload.getNumMessagesInBatch(), metadataAndPayload.readableBytes());

Will test the change tomorrow and raise a PR.

@merlimat merlimat added this to the 1.18 milestone Jun 14, 2017
@jai1 jai1 mentioned this issue Jun 14, 2017
Merged
jai1 pushed a commit to jai1/pulsar that referenced this issue Jun 15, 2017
Fixed message rate out with batches to count messages/s (apache#466)
067923c
jai1 added a commit that referenced this issue Jun 15, 2017
@jai1
Fixed message rate out with batches to count messages/s (#466) (#474)
042521a
hangc0276 pushed a commit to hangc0276/pulsar that referenced this issue May 26, 2021
@BewareMyPower
Optimize produce request handler (apache#466)
808b969 
### Motivation

There're some problems with produce performance.

The main problem is the pending produce queue. It holds multiple pending produce requests (`PendingProduce`), which wait until `PersistentTopic` is ready and the `MemoryRecords` is encoded. First, `PendingProduce`s wait for different futures of `PersistentTopic`. Second, encoding `MemoryRecords` is fast so that putting it to another thread is not necessary and could cause some performance overhead.

### Modifications

1. Encoding `MemoryRecords` in the same thread of `handleProduceRequest`.
2. Check if the `CompletableFuture<PersistentTopic>` is done.
   - If it's done, just publish the messages directly without pushing the pending produce requests to the queue.
   - Otherwise, reuse the previous `CompletableFuture<PersistentTopic>`. This trick is performed by `PendingTopicFutures`, which uses the previous `CompletableFuture<PersistentTopic>` by `thenApply` or `exceptionally`.
3. Add tests for `PendingTopicFutures`.
4. Use a map of partition and response instead of a map of partition and response future in `handleProduceRequest`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jai1 jai1

Labels
type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Milestone
1.18
Development

No branches or pull requests
2 participants
@merlimat @jai1