Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn #81374
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = None closed_at = <Date 2020-12-31.20:26:17.417> created_at = <Date 2019-06-07.11:53:39.380> labels = ['3.8', '3.9', '3.10', 'performance', '3.7', 'library'] title = 'Memory leak while running TCP/UDPServer with socketserver.ThreadingMixIn' updated_at = <Date 2021-03-11.12:01:19.341> user = 'https://github.com/maru-n'
activity = <Date 2021-03-11.12:01:19.341> actor = 'vstinner' assignee = 'none' closed = True closed_date = <Date 2020-12-31.20:26:17.417> closer = 'pablogsal' components = ['Library (Lib)'] creation = <Date 2019-06-07.11:53:39.380> creator = 'maru-n' dependencies =  files = ['48399'] hgrepos =  issue_num = 37193 keywords = ['patch', '3.7regression'] message_count = 23.0 messages = ['344926', '345606', '345796', '345815', '345817', '346413', '358133', '358457', '358647', '371339', '371431', '371433', '380168', '380172', '380209', '380212', '380217', '380237', '380354', '384136', '388101', '388104', '388503'] nosy_count = 8.0 nosy_names = ['jaraco', 'vstinner', 'ned.deily', 'martin.panter', 'pablogsal', 'miss-islington', 'maru-n', 'Wei Li'] pr_nums = ['13893', '23087', '23088', '23089', '23107', '23127', '24032', '24033', '24749', '24750'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'resource usage' url = 'https://bugs.python.org/issue37193' versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']
The text was updated successfully, but these errors were encountered:
UDP/TCPServer with socketserver.ThreadingMixin class (also ThreadingTCPServer and ThreadingUDPServer class) seems to be memory leak while running the server.
My code which wrote to check this is the following.
( I wrote this based on https://docs.python.org/3/library/socketserver.html#asynchronous-mixins)
Then I checked memory usage with profiling tool.
$ mprof run python mycode.py $ mprof plot
I attached result plot.
And also I checked this also more long time and I found memory usage was increased endlessly.
My environment is
Hardware: MacBook Pro (15-inch, 2018)
I guess it caused by a thread object is not released in spite of the thread finished to process request and thread object will be made infinitely until server_close() is called.
Looking at the code, this would be caused by bpo-31233. I expect 3.7+ is affected. 3.6 has similar code, but the leaking looks to be disabled by default. 2.7 doesn't collect a "_threads" list at all.
Looks like Victor was aware of the leak when he changed the code: <https://bugs.python.org/issue31233#msg304619\>, but maybe he pushed the code and then forgot about the problem.
A possible problem with Norihiro's solution is modifying the "_threads" list from multiple threads without any synchronization. (Not sure if that is a problem, or is it guaranteed to be okay due to GIL etc?) Also, since the thread is removing itself from the list, it will still run a short while after the removal, so there is a window when the "server_close" method will not wait for that thread. Might also defeat the "dangling thread" accounting that I believe was Victor's motivation for his change.
Wei's proposal is to check for cleaning up when a new request is handled. That relies on a new request coming in to free up memory. Perhaps we could use similar strategy to the Forking mixin, which I believe cleans up expired children periodically, without relying on a new request.
I marked bpo-37389 as a duplicate of this issue:
After putting a basic ThreadingUDPServer under load (500 messages per/second) I noticed that after a night it was consuming a lot of RAM given it does nothing with the data.
On inception, I noticed the _thread count inside the server was growing forever even though the sub-threads are done.
Setup a basic ThreadingUDPSever with handler that does nothing and check the request_queue_size, it seems to grow without limit.
The only way I could figure out to control it was to do this in a thread;
for thread in server._threads: # type: Thread if not thread.is_alive(): server._threads.remove(thread)
Shouldn't the server process do this when the thread is done?
Another workaround might be to set the new "block_on_close" flag (bpo-33540) to False on the server subclass or instance.
Victor: Replying to <https://bugs.python.org/issue37193#msg345817\> "What do I think of also using a weakref?", I assume you mean maintaining "_threads" as a WeakSet rather than a list object. That seems a nice way to solve the problem, but it seems redundant to me if other code such as Maru's proposal was also added to clean up the list.
Commit c415590 has introduced reference leaks:
Example buildbot failure:
As there is a release of 3.10 alpha 2 tomorrow I would be great if this could be fixed by tomorrow.
The change fixing a leak in socketserver introduces a leak in socketserver :-)
$ ./python -m test test_socketserver -u all -m test.test_socketserver.SocketServerTest.test_ThreadingTCPServer -R 3:3 0:00:00 load avg: 0.95 Run tests sequentially 0:00:00 load avg: 0.95 [1/1] test_socketserver beginning 6 repetitions 123456 ...... test_socketserver leaked [3, 3, 3] references, sum=9 test_socketserver leaked [3, 3, 3] memory blocks, sum=9 test_socketserver failed
== Tests result: FAILURE ==
1 test failed:
Total duration: 497 ms