Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Subscribers using TCP transport lack the appropriate backpressure and disconnect during network congestion #5494

Closed
Fidenz-Lasithh opened this issue Aug 13, 2021 · 23 comments · Fixed by #6066
Closed

Subscribers using TCP transport lack the appropriate backpressure and disconnect during network congestion #5494

Fidenz-Lasithh opened this issue Aug 13, 2021 · 23 comments · Fixed by #6066
Labels
BUG

Comments

@Fidenz-Lasithh
Copy link

Please describe your problem in detail, if necessary, you can upload the log file through the attachment:
I've an EMQX broker deployed on a Kubernetes cluster using Helm. I also have a separate set of publishers who publishes to this broker and two subscribers who has shared the subscription to the same topic.
The topic is consisted of 23 levels.

When the subscribers get around 4000 messages per second, MQTT clients get disconnected. Subscribers are configured to reconnect automatically. They get disconnected as soon as they get reconnected.

I'm wondering whether there is a configuration option in EMQX broker to overcome this.
@k32
Copy link
Contributor

k32 commented Aug 13, 2021

Hello,
Could you share emqx.log from one of the nodes? Also, which version of the broker are you running?

@Fidenz-Lasithh
Copy link
Author

The version I've used last is 4.3.7. But I experienced the same with 4.3.6

Following is extracted from 'kubectl logs'

EMQ X Broker 4.3.7 is running now!
2021-08-13T10:10:18.343037+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47568 [MQTT] socket_error: busy
2021-08-13T10:10:23.232754+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47616 [MQTT] socket_error: busy
2021-08-13T10:10:23.749960+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47622 [MQTT] socket_error: busy
2021-08-13T10:10:36.271755+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47704 [MQTT] socket_error: busy
2021-08-13T10:10:37.746636+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47746 [MQTT] socket_error: busy
2021-08-13T10:10:43.866436+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47838 [MQTT] socket_error: busy
2021-08-13T10:10:46.951663+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47876 [MQTT] socket_error: busy
2021-08-13T10:10:55.628198+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:47972 [MQTT] socket_error: busy
2021-08-13T10:10:56.961247+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:42668 [MQTT] socket_error: busy
2021-08-13T10:10:59.044766+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48036 [MQTT] socket_error: busy
2021-08-13T10:11:01.796577+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:42746 [MQTT] socket_error: busy
2021-08-13T10:11:02.917917+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48092 [MQTT] socket_error: busy
2021-08-13T10:11:06.711462+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48138 [MQTT] socket_error: busy
2021-08-13T10:11:07.481556+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48156 [MQTT] socket_error: busy
2021-08-13T10:11:11.965126+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:42898 [MQTT] socket_error: busy
2021-08-13T10:11:13.772165+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48248 [MQTT] socket_error: busy
2021-08-13T10:11:15.531373+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:42950 [MQTT] socket_error: busy
2021-08-13T10:11:23.242593+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43070 [MQTT] socket_error: busy
2021-08-13T10:11:25.527534+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43092 [MQTT] socket_error: busy
2021-08-13T10:11:29.144758+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43154 [MQTT] socket_error: busy
2021-08-13T10:11:32.517384+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43200 [MQTT] socket_error: busy
2021-08-13T10:11:33.583010+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43230 [MQTT] socket_error: busy
2021-08-13T10:11:36.776013+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43274 [MQTT] socket_error: busy
2021-08-13T10:11:47.366724+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48758 [MQTT] socket_error: busy
2021-08-13T10:11:55.374509+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43546 [MQTT] socket_error: busy
2021-08-13T10:11:58.997953+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:48942 [MQTT] socket_error: busy
2021-08-13T10:12:03.064607+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43682 [MQTT] socket_error: busy
2021-08-13T10:12:04.807164+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43708 [MQTT] socket_error: busy
2021-08-13T10:12:07.390150+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43744 [MQTT] socket_error: busy
2021-08-13T10:12:13.029809+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43830 [MQTT] socket_error: busy
2021-08-13T10:12:13.233499+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49158 [MQTT] socket_error: busy
2021-08-13T10:12:13.440279+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:43836 [MQTT] socket_error: busy
2021-08-13T10:12:19.538485+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49252 [MQTT] socket_error: busy
2021-08-13T10:12:26.603169+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:44028 [MQTT] socket_error: busy
2021-08-13T10:12:28.428509+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:44066 [MQTT] socket_error: busy
2021-08-13T10:12:30.332469+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49416 [MQTT] socket_error: busy
2021-08-13T10:12:32.395729+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:44118 [MQTT] socket_error: busy
2021-08-13T10:12:45.826957+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49650 [MQTT] socket_error: busy
2021-08-13T10:12:48.402606+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49690 [MQTT] socket_error: busy
2021-08-13T10:12:50.714503+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49720 [MQTT] socket_error: busy
2021-08-13T10:12:54.780455+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49780 [MQTT] socket_error: busy
2021-08-13T10:12:55.249598+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49784 [MQTT] socket_error: busy
2021-08-13T10:13:02.107246+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:49900 [MQTT] socket_error: busy
2021-08-13T10:13:14.478519+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50086 [MQTT] socket_error: busy
2021-08-13T10:13:32.800284+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50362 [MQTT] socket_error: busy
2021-08-13T10:13:34.059515+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50388 [MQTT] socket_error: busy
2021-08-13T10:13:37.423888+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50434 [MQTT] socket_error: busy
2021-08-13T10:13:47.726459+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:45272 [MQTT] socket_error: busy
2021-08-13T10:13:54.892073+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:45366 [MQTT] socket_error: busy
2021-08-13T10:13:55.862315+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50704 [MQTT] socket_error: busy
2021-08-13T10:13:59.900464+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50780 [MQTT] socket_error: busy
2021-08-13T10:14:06.836136+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:50884 [MQTT] socket_error: busy
2021-08-13T10:14:20.448610+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:45748 [MQTT] socket_error: busy
2021-08-13T10:14:21.890645+00:00 [warning] kafka-mqtt-source-cam-1-4b18b708-b0dc-4b16-a8b8-858b92e258f6@10.240.3.50:45770 [MQTT] socket_error: busy
2021-08-13T10:14:26.990455+00:00 [warning] kafka-mqtt-source-cam-0-b807b648-49a2-4375-949a-99ccc6ceddb3@10.240.3.53:51170 [MQTT] socket_error: busy
2021-08-13T10:48:47.033652+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:52512 [MQTT] socket_error: busy
2021-08-13T10:48:50.148500+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:52582 [MQTT] socket_error: busy
2021-08-13T10:48:56.034237+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40284 [MQTT] socket_error: busy
2021-08-13T10:48:56.516606+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:52672 [MQTT] socket_error: busy
2021-08-13T10:48:56.953298+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:52682 [MQTT] socket_error: busy
2021-08-13T10:49:04.257076+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:52858 [MQTT] socket_error: busy
2021-08-13T10:49:04.568681+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40456 [MQTT] socket_error: busy
2021-08-13T10:49:05.940696+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40478 [MQTT] socket_error: busy
2021-08-13T10:49:08.358415+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40522 [MQTT] socket_error: busy
2021-08-13T10:49:08.848200+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40528 [MQTT] socket_error: busy
2021-08-13T10:49:16.231810+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53066 [MQTT] socket_error: busy
2021-08-13T10:49:16.351373+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40662 [MQTT] socket_error: busy
2021-08-13T10:49:21.960476+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:40734 [MQTT] socket_error: busy
2021-08-13T10:49:23.979138+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:43278 [MQTT] socket_error: busy
2021-08-13T10:49:28.833826+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53220 [MQTT] socket_error: busy
2021-08-13T10:49:36.453711+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53408 [MQTT] socket_error: busy
2021-08-13T10:49:39.288585+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:40966 [MQTT] socket_error: busy
2021-08-13T10:49:40.780824+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53484 [MQTT] socket_error: busy
2021-08-13T10:49:42.945870+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53524 [MQTT] socket_error: busy
2021-08-13T10:49:44.842943+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53568 [MQTT] socket_error: busy
2021-08-13T10:49:45.691275+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41112 [MQTT] socket_error: busy
2021-08-13T10:49:48.096467+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41154 [MQTT] socket_error: busy
2021-08-13T10:49:48.560373+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53624 [MQTT] socket_error: busy
2021-08-13T10:49:49.934355+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41188 [MQTT] socket_error: busy
2021-08-13T10:49:50.820645+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53660 [MQTT] socket_error: busy
2021-08-13T10:49:51.083704+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53654 [MQTT] socket_error: busy
2021-08-13T10:49:57.152275+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41258 [MQTT] socket_error: busy
2021-08-13T10:49:57.368968+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53782 [MQTT] socket_error: busy
2021-08-13T10:49:59.998219+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53836 [MQTT] socket_error: busy
2021-08-13T10:50:01.182528+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:43768 [MQTT] socket_error: busy
2021-08-13T10:50:02.140802+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:53874 [MQTT] socket_error: busy
2021-08-13T10:50:03.673926+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41390 [MQTT] socket_error: busy
2021-08-13T10:50:03.833148+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:53878 [MQTT] socket_error: busy
2021-08-13T10:50:09.319710+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41490 [MQTT] socket_error: busy
2021-08-13T10:50:09.647878+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54016 [MQTT] socket_error: busy
2021-08-13T10:50:10.396656+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54032 [MQTT] socket_error: busy
2021-08-13T10:50:13.732666+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41560 [MQTT] socket_error: busy
2021-08-13T10:50:17.626947+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41666 [MQTT] socket_error: busy
2021-08-13T10:50:20.323795+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44042 [MQTT] socket_error: busy
2021-08-13T10:50:21.586093+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41734 [MQTT] socket_error: busy
2021-08-13T10:50:21.624960+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44066 [MQTT] socket_error: busy
2021-08-13T10:50:22.029491+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41742 [MQTT] socket_error: busy
2021-08-13T10:50:26.607648+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54360 [MQTT] socket_error: busy
2021-08-13T10:50:27.102655+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54368 [MQTT] socket_error: busy
2021-08-13T10:50:27.526669+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41848 [MQTT] socket_error: busy
2021-08-13T10:50:32.865018+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:41938 [MQTT] socket_error: busy
2021-08-13T10:50:33.556745+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44264 [MQTT] socket_error: busy
2021-08-13T10:50:34.634445+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:41970 [MQTT] socket_error: busy
2021-08-13T10:50:34.901262+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54530 [MQTT] socket_error: busy
2021-08-13T10:50:35.144969+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54532 [MQTT] socket_error: busy
2021-08-13T10:50:38.156457+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42042 [MQTT] socket_error: busy
2021-08-13T10:50:39.081527+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54628 [MQTT] socket_error: busy
2021-08-13T10:50:40.661436+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42082 [MQTT] socket_error: busy
2021-08-13T10:50:41.242373+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54672 [MQTT] socket_error: busy
2021-08-13T10:50:41.702097+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42104 [MQTT] socket_error: busy
2021-08-13T10:50:42.646728+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54700 [MQTT] socket_error: busy
2021-08-13T10:50:42.724753+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44418 [MQTT] socket_error: busy
2021-08-13T10:50:43.355401+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42150 [MQTT] socket_error: busy
2021-08-13T10:50:45.042374+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54748 [MQTT] socket_error: busy
2021-08-13T10:50:46.390128+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54776 [MQTT] socket_error: busy
2021-08-13T10:50:47.126218+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42224 [MQTT] socket_error: busy
2021-08-13T10:50:48.760954+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54830 [MQTT] socket_error: busy
2021-08-13T10:50:48.783741+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42254 [MQTT] socket_error: busy
2021-08-13T10:50:49.651760+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42270 [MQTT] socket_error: busy
2021-08-13T10:50:51.643754+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:54882 [MQTT] socket_error: busy
2021-08-13T10:50:51.954781+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42310 [MQTT] socket_error: busy
2021-08-13T10:50:52.443583+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42318 [MQTT] socket_error: busy
2021-08-13T10:50:53.220580+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54910 [MQTT] socket_error: busy
2021-08-13T10:50:54.697045+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:54954 [MQTT] socket_error: busy
2021-08-13T10:50:57.434533+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42418 [MQTT] socket_error: busy
2021-08-13T10:50:58.366934+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42428 [MQTT] socket_error: busy
2021-08-13T10:51:00.250957+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44696 [MQTT] socket_error: busy
2021-08-13T10:51:00.739251+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44710 [MQTT] socket_error: busy
2021-08-13T10:51:01.056888+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55084 [MQTT] socket_error: busy
2021-08-13T10:51:01.918155+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42498 [MQTT] socket_error: busy
2021-08-13T10:51:02.456895+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:55106 [MQTT] socket_error: busy
2021-08-13T10:51:03.160124+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44746 [MQTT] socket_error: busy
2021-08-13T10:51:04.342214+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42540 [MQTT] socket_error: busy
2021-08-13T10:51:05.172336+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42550 [MQTT] socket_error: busy
2021-08-13T10:51:06.336866+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:44796 [MQTT] socket_error: busy
2021-08-13T10:51:08.335860+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:55232 [MQTT] socket_error: busy
2021-08-13T10:51:09.925389+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42642 [MQTT] socket_error: busy
2021-08-13T10:51:13.010671+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42714 [MQTT] socket_error: busy
2021-08-13T10:51:13.669887+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42734 [MQTT] socket_error: busy
2021-08-13T10:51:14.013784+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42738 [MQTT] socket_error: busy
2021-08-13T10:51:14.743943+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42750 [MQTT] socket_error: busy
2021-08-13T10:51:16.164534+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42782 [MQTT] socket_error: busy
2021-08-13T10:51:21.144537+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55482 [MQTT] socket_error: busy
2021-08-13T10:51:22.621146+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55504 [MQTT] socket_error: busy
2021-08-13T10:51:24.843380+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:42932 [MQTT] socket_error: busy
2021-08-13T10:51:24.964533+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:42936 [MQTT] socket_error: busy
2021-08-13T10:51:28.207645+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55616 [MQTT] socket_error: busy
2021-08-13T10:51:29.005439+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:55626 [MQTT] socket_error: busy
2021-08-13T10:51:29.072760+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:43006 [MQTT] socket_error: busy
2021-08-13T10:51:29.257351+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45162 [MQTT] socket_error: busy
2021-08-13T10:51:31.384534+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:43054 [MQTT] socket_error: busy
2021-08-13T10:51:31.589054+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45202 [MQTT] socket_error: busy
2021-08-13T10:51:32.840841+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45230 [MQTT] socket_error: busy
2021-08-13T10:51:33.420302+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:43086 [MQTT] socket_error: busy
2021-08-13T10:51:36.370410+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45284 [MQTT] socket_error: busy
2021-08-13T10:51:38.427549+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45316 [MQTT] socket_error: busy
2021-08-13T10:51:38.792081+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:43186 [MQTT] socket_error: busy
2021-08-13T10:51:38.810653+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:43188 [MQTT] socket_error: busy
2021-08-13T10:51:38.995737+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45324 [MQTT] socket_error: busy
2021-08-13T10:51:39.915853+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45340 [MQTT] socket_error: busy
2021-08-13T10:51:41.233311+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:55850 [MQTT] socket_error: busy
2021-08-13T10:51:41.255577+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:43224 [MQTT] socket_error: busy
2021-08-13T10:51:44.661659+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45430 [MQTT] socket_error: busy
2021-08-13T10:51:45.749327+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55922 [MQTT] socket_error: busy
2021-08-13T10:51:45.793691+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45446 [MQTT] socket_error: busy
2021-08-13T10:51:48.971780+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:55952 [MQTT] socket_error: busy
2021-08-13T10:51:52.642792+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45560 [MQTT] socket_error: busy
2021-08-13T10:51:54.094949+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45588 [MQTT] socket_error: busy
2021-08-13T10:51:55.908283+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:56076 [MQTT] socket_error: busy
2021-08-13T10:51:59.154887+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:43482 [MQTT] socket_error: busy
2021-08-13T10:52:00.807814+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:56136 [MQTT] socket_error: busy
2021-08-13T10:52:00.926749+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45708 [MQTT] socket_error: busy
2021-08-13T10:52:04.388892+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45768 [MQTT] socket_error: busy
2021-08-13T10:52:07.778191+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:56246 [MQTT] socket_error: busy
2021-08-13T10:52:10.413617+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45860 [MQTT] socket_error: busy
2021-08-13T10:52:12.767578+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:43672 [MQTT] socket_error: busy
2021-08-13T10:52:13.725465+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45926 [MQTT] socket_error: busy
2021-08-13T10:52:16.775744+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:45960 [MQTT] socket_error: busy
2021-08-13T10:52:17.704867+00:00 [warning] kafka-mqtt-source-cam-0-d6b7b7f3-a570-4433-bc9d-3161ec3c26d3@10.240.0.150:56422 [MQTT] socket_error: busy
2021-08-13T10:52:19.327500+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46014 [MQTT] socket_error: busy
2021-08-13T10:52:19.697379+00:00 [warning] kafka-mqtt-source-cam-1-64ce7df2-3bb9-4a82-ad78-538e3c1c89ef@10.240.0.236:43776 [MQTT] socket_error: busy
2021-08-13T10:52:21.014664+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46038 [MQTT] socket_error: busy
2021-08-13T10:52:23.615242+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46090 [MQTT] socket_error: busy
2021-08-13T10:52:24.599812+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46112 [MQTT] socket_error: busy
2021-08-13T10:52:28.085970+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46160 [MQTT] socket_error: busy
2021-08-13T10:52:29.260331+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is activated, 84.32358221565107% cpu usage
2021-08-13T10:52:30.171205+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46198 [MQTT] socket_error: busy
2021-08-13T10:52:32.285611+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46236 [MQTT] socket_error: busy
2021-08-13T10:52:34.288951+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:43958 [MQTT] socket_error: busy
2021-08-13T10:52:43.861197+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46436 [MQTT] socket_error: busy
2021-08-13T10:52:46.242884+00:00 [warning] kafka-mqtt-source-cam-0-0ff0ca10-2c40-4e43-a4ed-7c65a7e70700@10.240.0.150:56880 [MQTT] socket_error: busy
2021-08-13T10:52:53.423297+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46586 [MQTT] socket_error: busy
2021-08-13T10:52:54.259620+00:00 [warning] kafka-mqtt-source-cam-1-ae878402-d6fa-4b11-8e48-f9ed5991aa36@10.240.0.236:44256 [MQTT] socket_error: busy
2021-08-13T10:53:00.747839+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46724 [MQTT] socket_error: busy
2021-08-13T10:53:03.277042+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46766 [MQTT] socket_error: busy
2021-08-13T10:53:03.790736+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46778 [MQTT] socket_error: busy
2021-08-13T10:53:04.621883+00:00 [warning] kafka-mqtt-source-cam-2-1f9e84a7-16db-46b8-acf0-efe346514913@10.240.3.115:46790 [MQTT] socket_error: busy
2021-08-13T10:54:29.262670+00:00 [warning] [Alarm Handler] Alarm high_cpu_usage is deactivated

@Fidenz-Lasithh
Copy link
Author

Hi @k32,
I've attached the entire file for your reference
log.txt

@k32
Copy link
Contributor

k32 commented Aug 13, 2021

Thanks for the information. Looking at the code, this warning message appears when the erlang port corresponding to the socket is busy, and it leads to closing of the socket.
Could you check CPU and memory usage of the EMQ X container? How many brokers do you have in the cluster?

@Fidenz-Lasithh
Copy link
Author

@k32,
I have a HPA created for the EMQ X broker and total memory usage is around 70%-80% and cpu usage is below 20%
For the above mentioned scenario there are 13 EMQ X pods running

@k32 k32 self-assigned this Aug 13, 2021
@k32
Copy link
Contributor

k32 commented Aug 13, 2021 •
edited

I see, thanks. Could you share some details about the configuration? Do you terminate SSL on a front proxy or the broker?

@Fidenz-Lasithh
Copy link
Author

Publishers are external to the cluster and they use the port 8883 with TLS and username password authentication.
Subscribers are internal to the cluster and they use the port 1883 without TLS or authentication..
Did I answer your question?

@Fidenz-Lasithh
Copy link
Author

UPDATE:
Publishing clients do not get disconnected. Only subscribing clients get disconnected.

@k32
Copy link
Contributor

k32 commented Aug 13, 2021

Thanks, this information adds up with what I see in the code: https://github.com/emqx/esockd/blob/master/src/esockd_transport.erl#L151

I'll check which options can be tuned for the tcp transport.

@Fidenz-Lasithh
Copy link
Author

@k32 ,
Until you come up with a solution, would you be able to let me know a version that does not have this bug?

@k32
Copy link
Contributor

k32 commented Aug 16, 2021 •
edited

Hello again,

I believe this problem is a result of network congestion. There is an alarm for that, could you check if you see it?

It looks like the TCP socket options are hardcoded in the code...
As a workaround you can try to increase listener.tcp.$name.sndbuf parameter or try to switch the subscribers to the SSL listener, because SSL transport uses a different approach to sending messages. It will cause increased CPU load, though.

@k32
Copy link
Contributor

k32 commented Aug 16, 2021 •
edited

Until you come up with a solution, would you be able to let me know a version that does not have this bug?

I will double-check if transport options have changed.

@k32
Copy link
Contributor

k32 commented Aug 16, 2021

@Fidenz-Lasithh I see that nosuspend option (that, I believe, can cause this error) was added to the esockd transport after v4.2.3. So you can try this version.

@Fidenz-Lasithh
Copy link
Author

@k32 ,
Thank you for the feedback! First I'll try the version 4.2.3 to check whether it works and get back to you. Turning on SSL will have a significant impact on the CPU usage.
Moving forward, will this be fixed in future versions?

@k32
Copy link
Contributor

k32 commented Aug 16, 2021

Moving forward, will this be fixed in future versions?

Absolutely, if my theory about network congestion is correct, then the current behavior of handing it is not correct.

@Fidenz-Lasithh
Copy link
Author

Fidenz-Lasithh commented Aug 16, 2021 •
edited

@k32 ,
Tested version 4.2.3 with 10,000 messages per second and it worked quite well. Had only a few disconnections.
I appreciate your support.

Moving forward, will this be fixed in future versions?

Absolutely, if my theory about network congestion is correct, then the current behavior of handing it is not correct.

It's nice to hear that. Would you be able to say in which version?

@k32
Copy link
Contributor

k32 commented Aug 16, 2021 •
edited

Would you be able to say in which version?

I will discuss it with the team. I think it deserves to be backported to the next 4.3.* version, as well as 5.0, of course.

@k32 k32 changed the title Subscriber keeps disconnecting when 'number of messages per second' increases Subscribers using TCP transport lack the appropriate backpressure and disconnect during network congestion Aug 17, 2021
@k32 k32 added BUG and removed Support labels Aug 17, 2021
@Fidenz-Lasithh
Copy link
Author

Hi @k32 ,
Is this fixed in version 4.3.8?

@zmstone
Copy link
Member

zmstone commented Sep 30, 2021

nosuspend was introduced mainly because we wanted to keep the socket owner responsive to management calls such as 'kick'
we'll perhaps change the 'kick' to 'kill' if timeout and remove nosuspension

k32 added a commit to k32/emqx that referenced this issue Nov 4, 2021
@k32
fix(emqx_connection): Introduce backpressure while sending data
27afecb 
Fixes emqx#5494
k32 added a commit to k32/emqx that referenced this issue Nov 4, 2021
@k32
fix(emqx_connection): Add backpressure to TCP connections
0a5a9bd 
Fixes emqx#5494
@k32 k32 mentioned this issue Nov 4, 2021
Merged
@k32 k32 closed this as completed in #6066 Nov 8, 2021
@Fidenz-Lasithh
Copy link
Author

Hi,
Is this fix included in version 4.3.10?

@zmstone
Copy link
Member

zmstone commented Nov 11, 2021

yes.

@Fidenz-Lasithh
Copy link
Author

@zmstone That sounds awesome. When can I expect 4.3.10 image to be available on docker hub?

@zmstone
Copy link
Member

zmstone commented Nov 12, 2021

Hi @Fidenz-Lasithh it's published, docker pull emqx/emqx:4.3.10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
BUG
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(emqx_connection): Add backpressure to TCP connections k32/emqx
3 participants
@zmstone @k32 @Fidenz-Lasithh