Reduction in Broker TPS on 2.8 Pulsar build #11639
Description
Hi All,
We have observed a reduction in Broker TPS on 2.8 builds compares to 2.6 build against the same test.
Environment Setup
Pulsar is setup in AWS account with following infrastructure configuration.
Note: If any changes done in infrastructure for a specific test are mentioned under performance test case details
| Component | Type | #Instances |
|---|---|---|
| Broker | c5.4xlarge | 3 |
| bookie | m5.xlarge | 6 |
| zookeeper | t2.large | 3 (One additional zk in east-1) |
| proxy | c5.2xlarge | 3 |
| manager | c5.2xlarge | 1 |
Config Customisations on build
Message(transparent) Encryption enabled
Logging enabled for Publish, Dispatch, ACK events.
TLS enabled over transport among all components.
Strict Authentication and authorization enabled at the proxy
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
managedLedgerDefaultAckQuorum=2
Pulsar 2.6 Perf test result
Swimlane capacity at ~60% CPU for 3KB msg size - Achieved TPS - 58k
3 proxies+3 zk+ 1 zk in east-1 6 bookies per region
3 Scripts, Each script has 3 instances, Topics:25, 7 inject producer users, 7 inject consumer users, 1 producer connections,1 consumer connections
Pulsar 2.8 Perf test result
Swimlane capacity at ~35% CPU for 3KB msg size - Achieved TPS - 37K
3 proxies+3 zk+ 1 zk in east-1 6 bookies per region
3 Scripts, Each script has 3 instances, Topics:25, 7 inject producer users, 7 inject consumer users, 1 producer connections,1 consumer connections
Observation :
Beyond one point broker TPS was not increasing beyond 35%.
We are suspecting there is some throttle value in broker configuration which is limiting broker CPU and TPS.