Nanomq process suspend after run and topic publish #1102

charlescui · 2023-03-16T04:39:24Z

Describe the bug
无论是DEB源安装，还是Docker安装，都有如下错误，引起Nanomq进程僵死，无响应，日志报错。

启动Nanomq服务
pub客户端连接并上报消息
Nanomq报错，并且无响应

以容器方式启动的命令如下:
docker run -d -p 1883:1883 -p 8883:8883 -v /etc/nanomq.conf:/etc/nanomq.conf --name nanomq emqx/nanomq

容器的id: 6c61ba6cbe74c57e4a34638f0228f509d70d5b336bf5912d8b6fa637cf7717a6

Expected behavior

错误日志如下:

2023-03-16 04:28:38 [1] WARN  /nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:450 nano_ctx_send: pipe 933801404 occupied! resending in cb!
2023-03-16 04:28:38 [1] WARN  /nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:450 nano_ctx_send: pipe 933801404 occupied! resending in cb!
2023-03-16 04:28:54 [1] WARN  /nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:450 nano_ctx_send: pipe 933801405 occupied! resending in cb!
2023-03-16 04:28:56 [1] WARN  /nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:605 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
2023-03-16 04:28:56 [1] WARN  /nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:853 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
2023-03-16 04:28:59 [1] WARN  /nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:450 nano_ctx_send: pipe 933801406 occupied! resending in cb!
2023-03-16 04:29:03 [1] WARN  /nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:605 tcptran_pipe_recv_cb: nni aio recv error!! Connection shutdown
2023-03-16 04:29:03 [1] WARN  /nanomq/nng/src/sp/transport/mqtt/broker_tcp.c:853 tcptran_pipe_recv_cb: tcptran_pipe_recv_cb: recv error rv: 139
2023-03-16 04:29:28 [1] WARN  /nanomq/nng/src/sp/protocol/mqtt/nmq_mqtt.c:396 nano_ctx_send: pipe id 933801404 is gone, pub failed

Actual Behavior

/etc/nanomq.conf配置文件如下:

# NanoMQ Configuration 0.14.0

# #============================================================
# # NanoMQ Broker
# #============================================================

system {
	daemon = false
	num_taskq_thread = 0
	max_taskq_thread = 0
	parallel = 0
}

mqtt.session {
	property_size = 32
	max_packet_size = 1024
	client_max_packet_size = 1024
	msq_len = 2048
	qos_duration = 10s
	keepalive_backoff = 1250
	allow_anonymous = true
}

listeners.tcp {
	enable = true
	bind = "0.0.0.0:1883"
}

listeners.ssl {
	enable = false
	bind = "0.0.0.0:8883"
	keyfile = "/etc/certs/key.pem"
	certfile = "/etc/certs/cert.pem"
	cacertfile = "/etc/certs/cacert.pem"
	verify_peer = false
	fail_if_no_peer_cert = false
}

listeners.ws {
	enable = true
	bind = "0.0.0.0:8083/mqtt"
}

http_server {
	enable = true
	port = 8081
	parallel = 2
	username = admin
	password = xxx
	auth_type = basic
	jwt {
		public.keyfile = "/etc/certs/jwt/jwtRS256.key.pub"
		private.keyfile = "/etc/certs/jwt/jwtRS256.key"
	}
}


log {
	to = [file, console]
	level = warn
	dir = "/tmp"
	file = "nanomq.log"
	rotation {
		size = 100MB
		count = 5
	}
}


sqlite {
	enable = true,
	disk_cache_size = 102400,
	mounted_file_path="/tmp/"
	flush_mem_threshold = 100
	resend_interval = 5000
}


webhook {
	enable = false
	url = "http://127.0.0.1:80"
	headers.content-type = "application/json"
	body.encoding = plain
	pool_size = 32
}

webhook.rule.client.connect = [{action: on_client_connect}]
webhook.rule.client.connack = [{action: on_client_connack}]
webhook.rule.client.connected = [{action: on_client_connected}]
webhook.rule.client.disconnected = [{action: on_client_disconnected}]
webhook.rule.client.subscribe = [{action: on_client_subscribe}]
webhook.rule.client.unsubscribe = [{action: on_client_unsubscribe}]
webhook.rule.session.subscribed = [{action: on_session_subscribed}]
webhook.rule.session.unsubscribed = [{action: on_session_unsubscribed}]
webhook.rule.session.terminated = [{action: on_session_terminated}]
webhook.rule.message.publish = [{action: on_message_publish}]
webhook.rule.message.delivered = [{action: on_message_delivered}]
webhook.rule.message.acked = [{action: on_message_acked}]


authorization {
	no_match = allow
	deny_action = ignore

	cache = {
		enable = false
		max_size = 32
		ttl = 1m
	}

	sources = [
		{
			type = simple
			enable = true

			users = [
				{
					username = "z2m"
					password = "xxx"
				}
				{
					username = "rule"
					password = "xxx"
				}
			]
		},
		{
			type = file
			enable = false

			rules = [
				## Allow MQTT client using username "dashboard"  to subscribe to "$SYS/#" topics
				{"permit": "allow", "username": "dashboard", "action": "subscribe", "topics": ["$SYS/#"]}

				## Deny "All Users" subscribe to "$SYS/#" "#" Topics
				{"permit": "deny", "username": "#", "action": "subscribe", "topics": ["$SYS/#", "#"]}

				## Allow any other publish/subscribe operation
				{"permit": "allow"}
			]
		},
		{
			type = http
			enable = false
			auth_req {
				url = "http://127.0.0.1:80/mqtt/auth"
				method = post

				headers.content-type = "application/x-www-form-urlencoded"

				params = {clientid = "%c", username = "%u", password = "%p"}
			}

			super_req {
				url = "http://127.0.0.1:80/mqtt/superuser"
				method = "post"
				headers.content-type = "application/x-www-form-urlencoded"
				params = {clientid = "%c", username = "%u", password = "%p"}
			}

			acl_req {
				url = "http://127.0.0.1:8991/mqtt/acl"
				method = "post"
				headers.content-type = "application/x-www-form-urlencoded"
				params = {clientid = "%c", username = "%u", access = "%A", ipaddr = "%a", topic = "%t", mountpoint = "%m"}
			}

			timeout = 5s
			connect_timeout = 5s
			pool_size = 32
		}
	]
}

bridges.mqtt {
	nodes = [
		{
			name = emqx
			enable = false
			connector {
				server = "mqtt-tcp://localhost:1883"
				proto_ver = 4
				username = user
				password = pwd
				clean_start = true
				keepalive = 60s
			}
			forwards = ["topic1/#", "topic2/#"]
			quic_keepalive = 120s
			quic_idle_timeout = 120s
			quic_discon_timeout = 20s
			quic_handshake_timeout = 60s
			hybrid_bridging = false
			congestion_control = cubic
			subscription = [
				{
					topic = "cmd/topic1"
					qos = 1
				},
				{
					topic = "cmd/topic2"
					qos = 2
				}
			]
            parallel = 2
            max_send_queue_len = 1024
            max_recv_queue_len = 1024

			ssl {
				enable = false
				keyfile = "/etc/certs/key.pem"
				certfile = "/etc/certs/cert.pem"
				cacertfile = "/etc/certs/cacert.pem"
			}
		}

	]
	sqlite {
		enable = false
		disk_cache_size = 102400
		mounted_file_path = "/tmp/"
		flush_mem_threshold = 100
		resend_interval = 5000
	}

}


bridges.aws.nodes = [
	{
		name = aws
		enable = false

		connector {
			server = "localhost:1883"
			proto_ver = 4
			username = user
			password = pwd
			clientid = "aws_bridge_client"
			clean_start = true
			keepalive = 60s
		}

		forwards = ["topic1/#", "topic2/#"]
		subscription = [
			{
				topic = "cmd/topic1"
				qos = 1
			},
			{
				topic = "cmd/topic2"
				qos = 2
			}
		]
		parallel = 2
		ssl {
			enable = false
			keyfile = "/etc/certs/key.pem"
			certfile = "/etc/certs/cert.pem"
			cacertfile = "/etc/certs/cacert.pem"
		}

	}
]


rules.option = OFF

rules.sqlite {
	path = "/tmp/sqlite_rule.db"
	enable = false
	rules = [
		{
			enable = true
			sql = "SELECT payload.x.y as y, payload.z as z FROM \"#\" WHERE y > 10 and z != 'str'"
			table = broker
		},
		{
			enable = true
			sql = "SELECT topic, payload FROM \"abc\""
			table = broker1
		},
		{
			enable = true
			sql = "SELECT * FROM \"#\" "
			table = broker2
		}
	]
}


rules.repub {
	enable = false
	rules = [
		{
			enable = true
			server = "mqtt-tcp://localhost:1883"
			topic = "topic/repub1"
			proto_ver = 4
			clientid = "repub_client1"
			keepalive = 60s
			clean_start = true
			username = username
			password = passwd
			sql = "SELECT payload.x.y as y, payload.z as z FROM \"#\" WHERE y > 10 and z != 'str'"
		},
		{
			enable = true
			server = "mqtt-tcp://localhost:1883"
			topic = "topic/repub2"
			proto_ver = 4
			clientid = "repub_client2"
			keepalive = 60s
			clean_start = true
			username = username
			password = passwd
			sql = "SELECT topic, payload FROM \"abc\""
		}
	]
}

rules.mysql {
	name = "mysql_rule_db"
	enable = false
	rules = [
		{
			enable = true
			table = broker
			host = localhost
			username = username
			password = password
			sql = "SELECT payload.x.y as y, payload.z as z FROM \"#\" WHERE y > 10 and z != 'str'"
		},
		{
			enable = true
			table = broker1
			host = localhost
			username = username
			password = password
			sql = "SELECT * FROM \"abc\""
		},
		{
			enable = true
			table = broker2
			host = localhost
			username = username
			password = password
			sql = "SELECT payload, qos FROM \"#\" WHERE qos > 0"
		}
	]
}

To Reproduce
If possible include actual reproduction test code here.
Minimal C test cases are perferred.

** Environment Details **

NanoMQ version
Operating system and version
Compiler and language used
testing scenario

Client SDK
If possible include the mqtt sdk you used to connect to nanomq
Minimal C test cases are perferred.

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

JaylinYu · 2023-03-16T06:54:03Z

Hello, I suspect the reason behind this is not nanomq but sth else... because I failed to reproduce your case, for NanoMQ 0.16.2, process works just fine in here. Plz try with strace or ptrace so that we can exclude other external issues.

Hi, 我怀疑您体验到僵死的原因并非来自于nanomq，而是其他因素。我不能搞根据你的配置文件重现您说的情况， log也只是普通的客户端链接断开所打印的信息。请问Nanomq的进程状态是？可否使用strace或者内置的ptrace启动来判断一下进程是否正常工作？或者开启桥接或trace级别的log看是否有其他运行信息打印

JaylinYu · 2023-03-24T06:43:25Z

gonna close this due to lack of feedback

JaylinYu added the Investigation quiestion remains ambiguous label Mar 16, 2023

JaylinYu added feedback Waiting for futher feedbacks and removed Investigation quiestion remains ambiguous labels Mar 24, 2023

JaylinYu closed this as completed Apr 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nanomq process suspend after run and topic publish #1102

Nanomq process suspend after run and topic publish #1102

charlescui commented Mar 16, 2023

JaylinYu commented Mar 16, 2023

JaylinYu commented Mar 24, 2023

Nanomq process suspend after run and topic publish #1102

Nanomq process suspend after run and topic publish #1102

Comments

charlescui commented Mar 16, 2023

JaylinYu commented Mar 16, 2023

JaylinYu commented Mar 24, 2023