Some sockets are opened but never read. UDP socket buffer memory usage increases

[nicoladefranceschi]

In the Zeroconf class init, some sockets:
https://github.com/jstasiak/python-zeroconf/blob/c7876108150cd251786db4ab52dadd1b2283d262/zeroconf.py#L1810

are created in both the unicast and multicast case.

But here:
https://github.com/jstasiak/python-zeroconf/blob/c7876108150cd251786db4ab52dadd1b2283d262/zeroconf.py#L1858-L1862
in case of multicast, the _respond_sockets are never read.

This causes some problem with the Recv queue of UDP, because the OS (in my case Ubuntu) keeps all the packets in memory waiting for the socket to read them, but this never happens and the memory keeps growing "forever"!

This is my output of sudo ss -nlpu:

State     Recv-Q     Send-Q                                Local Address:Port            Peer Address:Port                                                     
UNCONN    156416     0                                           0.0.0.0:5353                 0.0.0.0:*         users:(("python",pid=10571,fd=54))             

To solve:
I tried running these two lines
https://github.com/jstasiak/python-zeroconf/blob/c7876108150cd251786db4ab52dadd1b2283d262/zeroconf.py#L1861-L1862
even in the case of multicast.
The problem is "solved".

I'm sure that is not the right solution and may actually be logically wrong doing that for this case.
I'm not an expert, but maybe there is a way to actually tell the OS that those sockets are not interested in listening for packets.

[psidlo]

@nicoladefranceschi I have the same issue. May you please provide be more specific how did you hotfixed it? Or provide hotfix version of your zeroconf.py?

[kdvlr]

I tried using @nicoladefranceschi's suggestion, however, while the UDP receive packets issue seems to be mitigiated, there is a CPU spike whenever the sockets are cleaned up which is causing other services to fail.

The problem is zeroconf seems to be attaching to all the interfaces. If the host is a docker host, then obviously there are multiple interfaces and zeroconf is attaching to all of them which then leads to UDP memory continuously increasing and eventually crashing.

[meme1337]

I have the same issue that is being reported: docker host saturating memory because zeroconf is attaching to all interfaces.

[kdvlr]

I solved the problem by using macvlan on my docker container - https://blog.oddbit.com/post/2018-03-12-using-docker-macvlan-networks/

[meme1337]

Thanks @kdvlr!
I will definitely look into that.

[fireinice]

i use macvlan as @kdvlr suggestion, but still bind to five 5353 port while only read one.

[meme1337]

i use macvlan as @kdvlr suggestion, but still bind to five 5353 port while only read one.

Sorry for the late reply. I haven’t solved this issue as well and I’m wondering if this happens also on other installations, but is simply ignored; or if it’s something that can be solved by configuring properly zeroconf, but I doubt it.

[meme1337]

Bump.
I didn't manage to solve this issue. Are there any updates on this?

[ioerror]

The recvq issue hit a few of my programs as well. I'd be willing to write a fix for zeroconf if the authors would be interested in and willing to merge a fix. Here's some code that should fix it for users of the zeroconf library if upstream doesn't fix the problem or until they do release a fix for this issue.

For ServiceBrowsers:

zeroconf = Zeroconf()
# This is a workaround for a memory leak that exists in the zeroconf library
zeroconf_fix_memleak(zeroconf)
browser = ServiceBrowser(zeroconf, _LABEL, YourCustomServiceListener())
...

For publishing services:

zeroconf = Zeroconf(interfaces=interfaces)
zeroconf.register_service(service_info)
# This is a workaround for a memory leak that exists in the zeroconf library
zeroconf_fix_memleak(zeroconf)
...

The zeroconf_fix_memleak code needs the following:

from logging import getLogger
from zeroconf import _MAX_MSG_ABSOLUTE

class FixMemoryLeak(object):
    """
    See https://github.com/jstasiak/python-zeroconf/issues/171 to understand
    why we need this hack.
    """

    def __init__(self):
        self.log = getLogger()

    def handle_read(self, socket_):
        try:
            data, (addr, port) = socket_.recvfrom(_MAX_MSG_ABSOLUTE)
            self.log.debug('FixMemoryLeak: %s', data)
            return
        except Exception as ex:
            self.log.debug('FixMemoryLeak exception: %s', ex)
            return

def zeroconf_fix_memleak(zeroconf):
    """
    This is a workaround for a memory leak that exists in the zeroconf library.
    """
    log = getLogger()
    for s in zeroconf._respond_sockets:
        log.debug('zeroconf leaky socket: %s', s)
        zeroconf.engine.add_reader(FixMemoryLeak(), s)

This hacky fix is released under the same license as python-zeroconf.

[emontnemery]

@ioerror please open a PR if you have a fix in mind👍

[jstasiak]

Agreed, a fix would be most appreciated.

[jstasiak]

All: #188 (and rebased version, #270) is likely to resolve this.

[jstasiak]

#270 is almost ready to be merged (I think specifying IPv6 interfaces by index may be broken now but I'll verify) – I'd appreciate if someone verified that that PR (branch rebase-188) works for them without regressions.