Skip to content

The maximum number of outstanding messages (--max-outstanding) configuration may be inaccurate when the topic is multi-partitioned #4159

@kuangye098

Description

@kuangye098

In the Producer Performance Benchmark, Topic has a partition number of 1024, specifying the parameter [-o 20000], and found that the maximum number of outstanding messages per producer is 48.

02:23:10.036 [pulsar-client-io-2-1] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: {
  "topicName" : "persistent://xxxx/internal/benchmark2",
  "producerName" : null,
  "sendTimeoutMs" : 0,
  "blockIfQueueFull" : true,
  "maxPendingMessages" : 48,
  "maxPendingMessagesAcrossPartitions" : 50000,
  "messageRoutingMode" : "RoundRobinPartition",
  "hashingScheme" : "JavaStringHash",
  "cryptoFailureAction" : "FAIL",
  "batchingMaxPublishDelayMicros" : 1000,
  "batchingMaxMessages" : 1000,
  "batchingEnabled" : true,
  "compressionType" : "NONE",
  "initialSequenceId" : null,
  "autoUpdatePartitions" : true,
  "properties" : { }
}
  1. Created a Topic with 1024 partitions.
  2. Test with the following command:
/pulsar-perf produce persistent://xxxx/internal/benchmark2 -m 10000000 -n 10 -i 5 -time 600 -o 20000 -u pulsar://10.0.231.202:6650,10.0.219.35:6650,10.0.250.94:6650> 0428-002-producer.log

Expected
The number of outstanding consumption messages per partition is 20000 or 20000/partition number.

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/bugThe PR fixed a bug or issue reported a bug

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions