Can provide p99<=1.5ms ?

[Cczzzz]

How can I modify the configuration or parameters to achieve this indicator。

[RongtongJin]

You can try the default configuration of version 4.9.1 first.
Latency is related to many aspects, including network quality and TPS (latency is inversely proportional to TPS during stress testing). It is recommended to look at it as a whole

[Cczzzz]

I try 4.9.1 and jdk 11 zgc。 in jdk 11 zgc is amazing ， gc stw in 1ms ，avg 0.4ms 。
But the server is delayed p99 at least 3-4ms

[areyouok]

You should check if there is bottleneck in client side.

I setup a 4.9.1 master/slave pair. 6 client send total 95000 TPS (produce and consume). The P99 of produce is 1.0 ms.

All configuration is default.

# sudo sysctl -w vm.min_free_kbytes=10000000 # modify this carefully
sudo sysctl -w vm.overcommit_memory=1
sudo sysctl -w vm.drop_caches=1
sudo sysctl -w vm.zone_reclaim_mode=0
sudo sysctl -w vm.max_map_count=655360
sudo sysctl -w vm.dirty_background_ratio=50
sudo sysctl -w vm.dirty_ratio=50
sudo sysctl -w vm.dirty_writeback_centisecs=360000
sudo sysctl -w vm.page-cluster=3
sudo sysctl -w vm.swappiness=1

[Cczzzz]

You should check if there is bottleneck in client side.

I setup a 4.9.1 master/slave pair. 6 client send total 95000 TPS (produce and consume). The P99 of produce is 1.0 ms.

All configuration is default.

# sudo sysctl -w vm.min_free_kbytes=10000000 # modify this carefully
sudo sysctl -w vm.overcommit_memory=1
sudo sysctl -w vm.drop_caches=1
sudo sysctl -w vm.zone_reclaim_mode=0
sudo sysctl -w vm.max_map_count=655360
sudo sysctl -w vm.dirty_background_ratio=50
sudo sysctl -w vm.dirty_ratio=50
sudo sysctl -w vm.dirty_writeback_centisecs=360000
sudo sysctl -w vm.page-cluster=3
sudo sysctl -w vm.swappiness=1

it is on DLedger ？DLedger performance is lower than master/slave

[areyouok]

You should check if there is bottleneck in client side.
I setup a 4.9.1 master/slave pair. 6 client send total 95000 TPS (produce and consume). The P99 of produce is 1.0 ms.
All configuration is default.

# sudo sysctl -w vm.min_free_kbytes=10000000 # modify this carefully
sudo sysctl -w vm.overcommit_memory=1
sudo sysctl -w vm.drop_caches=1
sudo sysctl -w vm.zone_reclaim_mode=0
sudo sysctl -w vm.max_map_count=655360
sudo sysctl -w vm.dirty_background_ratio=50
sudo sysctl -w vm.dirty_ratio=50
sudo sysctl -w vm.dirty_writeback_centisecs=360000
sudo sysctl -w vm.page-cluster=3
sudo sysctl -w vm.swappiness=1

it is on DLedger ？DLedger performance is lower than master/slave

no

[Cczzzz]

I add Metrics and recompile。write page cache is quickly。the part of raft sync is bottleneck。
specifically is ：

 long now = this.defaultMessageStore.getSystemClock().now();
    return dledgerFuture.thenApply(appendEntryResponse -> { 
        metrics.putRtRAFT(this.defaultMessageStore.getSystemClock().now() - now);//my Metrics
        PutMessageStatus putMessageStatus = PutMessageStatus.UNKNOWN_ERROR;
        switch (DLedgerResponseCode.valueOf(appendEntryResponse.getCode())) {
            case SUCCESS:
                putMessageStatus = PutMessageStatus.PUT_OK;
                break;
            case INCONSISTENT_LEADER:
            case NOT_LEADER:
            case LEADER_NOT_READY:
            case DISK_FULL:
                putMessageStatus = PutMessageStatus.SERVICE_NOT_AVAILABLE;
                break;
            case WAIT_QUORUM_ACK_TIMEOUT:
                //Do not return flush_slave_timeout to the client, for the ons client will ignore it.
                putMessageStatus = PutMessageStatus.OS_PAGECACHE_BUSY;
                break;
            case LEADER_PENDING_FULL:
                putMessageStatus = PutMessageStatus.OS_PAGECACHE_BUSY;
                break;
        }
        PutMessageResult putMessageResult = new PutMessageResult(putMessageStatus, appendResult);
        if (putMessageStatus == PutMessageStatus.PUT_OK) {
            // Statistics
            storeStatsService.getSinglePutMessageTopicTimesTotal(finalTopic).incrementAndGet();
            storeStatsService.getSinglePutMessageTopicSizeTotal(msg.getTopic()).addAndGet(appendResult.getWroteBytes());
        }
        return putMessageResult;
    });

wait dledgerFuture complete。I looked at the code inside ，all actions is asynch，and use completableFuture。

this metrics result：
count = 30686435
min = 0
max = 4
mean = 1.40
stddev = 0.64
median = 1.00
75% <= 2.00
95% <= 2.00
98% <= 2.00
99% <= 2.00
99.9% <= 4.00

4.9.1、 jdk 11 、zgc.、and tps 1.8w ,msgsize 1k. and also execute os.sh.

producer avg ms 1.7ms，remove the network delay of 0.3ms. so raft is bottleneck

[areyouok]

In my test the network ping latency is 0.06ms

[Cczzzz]

In my test the network ping latency is 0.06ms
I tested the master-slave mode.It is very easy to p99<1ms. The gap is too big of raft. why ?

[ShannonDing]

link #3368