windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。

[linshengbin]

windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。

[HerrMuellerluedenscheid]

From google translate: Under Windows, the memory occupied by the aqmtt service can be automatically released, but Linux has not been able to release it.

@linshengbin can you provide some more detail or information on the issue?

[linshengbin]

压测环境： python 3.4.3 压测记录： linux上起了broker。（Linux (none) 3.10.108-rt80-Cavium-Octeon+ #1 SMP Mon Aug 9 21:35:34 CST 2021 mips64 GNU/Linux） 然后windows启用了500个客户端id，进行client_publish循环压测。 VmRSS内存从20MB到100MB，等待一段时间（12小时左右）后，发现linux上的内存占用一直不释放，还是保持100088kb。 但windows端起的broker，内存降至18552KB。

…

------------------ 原始邮件 ------------------ 发件人: "Yakifo/amqtt" ***@***.***>; 发送时间: 2021年9月24日(星期五) 下午4:35 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Yakifo/amqtt] windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。 (#75) From google translate: Under Windows, the memory occupied by the aqmtt service can be automatically released, but Linux has not been able to release it. @linshengbin can you provide some more detail or information on the issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[sjlongland]

python 3.4.3

Wow, I don't think amqtt is going to work at all on such an old release of Python with its heavy-use of type annotations and async/await.

[linshengbin]

***@***.***/yield from。因为HBMQTT开源项目的社区已经没有更新了。所以只能通过这边来问问题了。

…

------------------ 原始邮件 ------------------ 发件人: "Yakifo/amqtt" ***@***.***>; 发送时间: 2021年9月26日(星期天) 下午4:54 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Yakifo/amqtt] windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。 (#75) python 3.4.3 Wow, I don't think amqtt is going to work at all on such an old release of Python with its heavy-use of type annotations and async/await. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[sjlongland]

(via Google Translate)

@.***/yield from. Because the HBMQTT open source project community has not been updated. So I can only ask questions through this side.

Yeah, it's fair to say, hbmqtt is pretty much history now. Of course, it's an open-source project, as is amqtt; so nothing stopping you or anybody else forking the code and fixing it, but I wouldn't hold my breath on fixes to amqtt being backported to hbmqtt.

Linux (none) 3.10.108-rt80-Cavium-Octeon+

That implies this is some sort of embedded device (router perhaps) … maybe the build environment needs re-jigging to do it, but python3.7 does work in such environments.

[FlorianLudwig]

@linshengbin could you share the code you use for load-testing?

So we could reproduce it

[linshengbin]

谢谢您的回复。我继续在Ubuntu的环境下测试高版本python特性的amqtt，看看是否能释放内存。 您之前有这样测试过吗？开500个客户端，持续测试一段时间。对比刚开启用amqtt broker时的内存占用对比。内存释放情况如何？

…

------------------ 原始邮件 ------------------ 发件人: "Yakifo/amqtt" ***@***.***>; 发送时间: 2021年9月26日(星期天) 下午5:24 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Yakifo/amqtt] windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。 (#75) (via Google Translate) @.***/yield from. Because the HBMQTT open source project community has not been updated. So I can only ask questions through this side. Yeah, it's fair to say, hbmqtt is pretty much history now. Of course, it's an open-source project, as is amqtt; so nothing stopping you or anybody else forking the code and fixing it, but I wouldn't hold my breath on fixes to amqtt being backported to hbmqtt. Linux (none) 3.10.108-rt80-Cavium-Octeon+ That implies this is some sort of embedded device (router perhaps) … maybe the build environment needs re-jigging to do it, but python3.7 does work in such environments. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[linshengbin]

import logging import asyncio from amqtt.client import MQTTClient, ConnectException logger = logging.getLogger(__name__) async def t_coro(): while True: ss = "CLIENT_ID_" for i in range(0, 500): try: C = MQTTClient(client_id=ss + str(i)) await C.connect("mqtt://172.30.81.246:1883/") await C.publish("a/b", b"TEST MESSAGE WITH QOS_0", qos=0x00) await C.publish("a/b", b"TEST MESSAGE WITH QOS_1", qos=0x01) await C.publish("a/b", b"TEST MESSAGE WITH QOS_2", qos=0x02) await C.disconnect() except ConnectException as ce: logger.error("Connection failed: %s" % ce) # asyncio.get_event_loop().stop() if __name__ == "__main__": formatter = ("[%(asctime)s] %(name)s {%(filename)s:%(lineno)d} %(levelname)s - %(message)s") logging.basicConfig(level=logging.ERROR, format=formatter) asyncio.get_event_loop().run_until_complete(t_coro())

…

------------------ 原始邮件 ------------------ 发件人: "Yakifo/amqtt" ***@***.***>; 发送时间: 2021年9月26日(星期天) 下午5:49 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Yakifo/amqtt] windows下，aqmtt服务占用的内存能自动释放，但linux一直释放不了。 (#75) @linshengbin could you share the code you use for load-testing? So we could reproduce it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[sjlongland]

Also, small tip (which applies to amqtt in general too, I see the pattern there in lots of places):

            try:
                # … etc …
                pass
            except ConnectException as ce:
                logger.error("Connection failed: %s" % ce)

Not the best way to log an exception for two reasons:

1. % always formats the string, whether logger is configured to emit logs at error level or not
2. you miss out on the stack trace, which can be real handy for troubleshooting

logger.error("Connection failed: %s", ce) addresses (1) whilst ignoring (2). The logging module has the exc_info keyword argument, which pretty much will do this for you… I'd write the above as:

            try:
                # … etc …
                pass
            except ConnectException:
                logger.error("Connection failed", exc_info=True)

Or, for error, there's a shortcut:

            try:
                # … etc …
                pass
            except ConnectException:
                logger.exception("Connection failed")