Skip to content

Commit

Permalink
Add benchmark-testing.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@rmqtt
rmqtt committed Sep 6, 2023
1 parent 65833bb commit 8105220
Showing 1 changed file with 328 additions and 0 deletions.
328 changes: 328 additions & 0 deletions docs/zh_CN/benchmark-testing.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,328 @@
[English](../en_US/benchmark-testing.md) | 简体中文


# 软硬件环境
| 项目 | 内容 | |
|-------------|-------------------------------------------|-----------------------------------------------------------------|
| 操作系统 | x86_64 GNU/Linux | Rocky Linux 9.2 (Blue Onyx) |
| CPU | Intel(R) Xeon(R) CPU E5-2696 v3 @ 2.30GHz | 72(CPU(s)) = 18(Core(s)) * 2(Thread(s) per core) * 2(Socket(s)) |
| 内存 | DDR3/2333 | 128G |
| 磁盘 | | 2T |
| 容器 | podman | v4.4.1 |
| 测试客户端 | docker.io/rmqtt/rmqtt-bench:latest | v0.1.3 |
| MQTT Broker | docker.io/rmqtt/rmqtt:latest | v0.2.16 |
| 其它 | 测试客户端和MQTT Broker同服 | |


# 单机模式

Broker主要配置(rmqtt.toml):
```
##--------------------------------------------------------------------
## Plugins
##--------------------------------------------------------------------
#Plug in configuration file directory
plugins.dir = "/app/rmqtt/etc/plugins"
#Plug in started by default, when the mqtt server is started
plugins.default_startups = [
#"rmqtt-cluster-broadcast",
#"rmqtt-cluster-raft",
#"rmqtt-auth-http",
#"rmqtt-web-hook",
#"rmqtt-retainer",
"rmqtt-http-api"
]
##--------------------------------------------------------------------
## Listeners
##--------------------------------------------------------------------
##--------------------------------------------------------------------
## MQTT/TCP - External TCP Listener for MQTT Protocol
listener.tcp.external.addr = "0.0.0.0:1883"
listener.tcp.external.workers = 68
listener.tcp.external.max_connections = 1024000
listener.tcp.external.max_handshaking_limit = 1500 #单机设置为1500，集群版本设置为500
listener.tcp.external.handshake_timeout = "30s"
listener.tcp.external.max_packet_size = "1m"
listener.tcp.external.backlog = 1024
listener.tcp.external.allow_anonymous = true
listener.tcp.external.allow_zero_keepalive = true
listener.tcp.external.min_keepalive = 0
listener.tcp.external.max_keepalive = 65535
listener.tcp.external.keepalive_backoff = 0.75
listener.tcp.external.max_inflight = 16
listener.tcp.external.max_mqueue_len = 1000
listener.tcp.external.mqueue_rate_limit = "1000,1s"
listener.tcp.external.max_clientid_len = 65535
listener.tcp.external.max_qos_allowed = 2
listener.tcp.external.max_topic_levels = 0
listener.tcp.external.retain_available = true
listener.tcp.external.session_expiry_interval = "2h"
listener.tcp.external.message_retry_interval = "20s"
listener.tcp.external.message_expiry_interval = "5m"
listener.tcp.external.max_subscriptions = 0
listener.tcp.external.shared_subscription = true
listener.tcp.external.max_topic_aliases = 32
```

单机Broker端podman命令:
```
podman run -d --name rmqtt1 --privileged --ulimit nofile=1020000 --network="compose_rmqtt_bridge1" --network-alias="node12.rmqtt.io" -p 1884:1883 -p 8884:8883 -p 21884:11883 -p 7064:6060 -v /home/dev/app/compose/log/1:/var/log/rmqtt -v /home/dev/app/compose/etc:/app/rmqtt/etc rmqtt/rmqtt:latest -f ./etc/rmqtt.toml --id 1
```

生成单机订阅客户端podman-compose配置脚本:

build-bench-sub-single.sh
```
#!/bin/bash
n=$1
start=$2
echo "version: '3'
services :"
for ((i=$start; i<$start+$n; i++))
do
echo "
mqtt-bench-a$i:
image: rmqtt/rmqtt-bench:latest
restart: on-failure:3
command: v3 -C -c 25000 -E \"sub-a$i-{no}\" -S -t iot/a$i-{no} --addrs \"node12.rmqtt.io:1883\" --ifaddrs \"a$i.rmqtt-bench.io\"
networks:
rmqtt_bridge1:
aliases:
- a$i.rmqtt-bench.io"
done
echo "networks:
rmqtt_bridge1:
driver: bridge"
```


创建docker-compose.yaml
```
sh build-bench-sub-single.sh 10 1 > docker-compose-bench-subs-single-10-1.yaml
sh build-bench-sub-single.sh 10 11 > docker-compose-bench-subs-single-10-11.yaml
sh build-bench-sub-single.sh 10 21 > docker-compose-bench-subs-single-10-21.yaml
sh build-bench-sub-single.sh 10 31 > docker-compose-bench-subs-single-10-31.yaml
```
一个容器创建25000个连接，一个文件10个容器，总的可以创建100万连接。

创建bridge网络：
```
podman network create compose_rmqtt_bridge1
```

## 连接并发性能

连接成功后会同时发起订阅。分4批次建立连接，一批次25万客户端同时发起连接请求。

| 项目 | 值 | 说明 |
|-------------|---------------|----------------|
| 并发客户端总数 | 100万 | 25000 * 10 * 4 |
| 连接握手速率 | (5500-7000)/秒 | 平均速率 |
| Broker进程内存 | 75.4G | |
| Broker进程CPU | (1-200)% | |
| Bench-subs进程内存 | 400M * 40 | |
| Bench-subs进程CPU | (0-20)% * 40 | |

查看状态:
```
curl "http://127.0.0.1:7064/api/v1/stats" | jq
[
{
"node": {
"id": 1,
"name": "1@127.0.0.1",
"status": "Running"
},
"stats": {
"connections.count": 1000000,
"connections.max": 1000000,
"handshakings.count": 0,
"handshakings.max": 62,
"handshakings_active.count": 0,
"handshakings_rate.count": 7282.29,
"handshakings_rate.max": 16600.09,
"inflights.count": 0,
"inflights.max": 0,
"message_queues.count": 0,
"message_queues.max": 0,
"retained.count": 0,
"retained.max": 0,
"routes.count": 1000000,
"routes.max": 1000000,
"sessions.count": 1000000,
"sessions.max": 1000000,
"subscriptions.count": 1000000,
"subscriptions.max": 1000000,
"subscriptions_shared.count": 0,
"subscriptions_shared.max": 0,
"topics.count": 1000000,
"topics.max": 1000000
}
}
]
```

## 消息吞吐性能

生成单机发布客户端podman-compose配置脚本:

build-bench-pub-single.sh
```
#!/bin/bash
n=$1
start=$2
echo "version: '3'
services :"
for ((i=$start; i<$start+$n; i++))
do
echo "
mqtt-bench-pub-a$i:
image: rmqtt/rmqtt-bench:latest
restart: on-failure:3
command: v3 -C -c 1 -E \"pub-a$i-{no}\" -P -t iot/a$i-{no} -R 0 24999 -I 0 --addrs \"node12.rmqtt.io:1883\" --ifaddrs \"a$i.rmqtt-bench-pub.io\"
networks:
rmqtt_bridge1:
aliases:
- a$i.rmqtt-bench-pub.io"
done
echo "networks:
rmqtt_bridge1:
driver: bridge"
```

创建docker-compose.yaml
```
sh build-bench-pub-single.sh 40 1 > docker-compose-bench-pubs-single-40-1.yaml
```
一个容器创建1个连接随机向25000个订阅端循环发送消息，总共创建40连接。


| 项目 | 值 | 说明 |
|-----------------|---------------|----------------|
| 订阅客户端数量 | 100万 | |
| 发布客户端数量 | 40 | |
| 消息吞吐速率 | 15万/秒 | |
| Broker进程内存 | 88.2G | |
| Broker进程CPU | (2600-2800)% | |
| Bench-subs进程内存 | 430M * 40 | |
| Bench-subs进程CPU | (20-30)% * 40 | |
| Bench-pubs进程内存 | 16M * 40 | |
| Bench-pubs进程CPU | (20-30)% * 40 | |

查看状态:
```
curl "http://127.0.0.1:7064/api/v1/stats" | jq
[
{
"node": {
"id": 1,
"name": "1@127.0.0.1",
"status": "Running"
},
"stats": {
"connections.count": 1000040,
"connections.max": 1000040,
"handshakings.count": 0,
"handshakings.max": 67,
"handshakings_active.count": 0,
"handshakings_rate.count": 399.69,
"handshakings_rate.max": 25261.64,
"inflights.count": 5480,
"inflights.max": 19407,
"message_queues.count": 1,
"message_queues.max": 33,
"retained.count": 0,
"retained.max": 0,
"routes.count": 1000000,
"routes.max": 1000000,
"sessions.count": 1000040,
"sessions.max": 1000040,
"subscriptions.count": 1000000,
"subscriptions.max": 1000000,
"subscriptions_shared.count": 0,
"subscriptions_shared.max": 0,
"topics.count": 1000000,
"topics.max": 1000000
}
}
]
```

```
date & curl "http://127.0.0.1:7064/api/v1/metrics" | jq | grep messages
Wed Sep 6 21:07:15 CST 2023
"messages.acked": 1321545894,
"messages.acked.admin": 0,
"messages.acked.custom": 1321545895,
"messages.acked.lastwill": 0,
"messages.acked.retain": 0,
"messages.acked.system": 0,
"messages.delivered": 1321545978,
"messages.delivered.admin": 0,
"messages.delivered.custom": 1321545979,
"messages.delivered.lastwill": 0,
"messages.delivered.retain": 0,
"messages.delivered.system": 0,
"messages.dropped": 0,
"messages.nonsubscribed": 0,
"messages.nonsubscribed.admin": 0,
"messages.nonsubscribed.custom": 0,
"messages.nonsubscribed.lastwill": 0,
"messages.nonsubscribed.system": 0,
"messages.publish": 1321546048,
"messages.publish.admin": 0,
"messages.publish.custom": 1321546048,
"messages.publish.lastwill": 0,
"messages.publish.system": 0,
[1]+ Done date
```

```
date & curl "http://127.0.0.1:7064/api/v1/metrics" | jq | grep messages
Wed Sep 6 21:07:54 CST 2023
"messages.acked": 1327420995,
"messages.acked.admin": 0,
"messages.acked.custom": 1327420995,
"messages.acked.lastwill": 0,
"messages.acked.retain": 0,
"messages.acked.system": 0,
"messages.delivered": 1327421038,
"messages.delivered.admin": 0,
"messages.delivered.custom": 1327421039,
"messages.delivered.lastwill": 0,
"messages.delivered.retain": 0,
"messages.delivered.system": 0,
"messages.dropped": 0,
"messages.nonsubscribed": 0,
"messages.nonsubscribed.admin": 0,
"messages.nonsubscribed.custom": 0,
"messages.nonsubscribed.lastwill": 0,
"messages.nonsubscribed.system": 0,
"messages.publish": 1327421058,
"messages.publish.admin": 0,
"messages.publish.custom": 1327421060,
"messages.publish.lastwill": 0,
"messages.publish.system": 0,
[1]+ Done date
```

# Raft集群模式


## 连接并发性能

0 comments on commit 8105220

Please sign in to comment.