New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

multidict 4.5 memory leak #307

Closed
byashimov opened this Issue Nov 21, 2018 · 15 comments

Comments

Projects
None yet
6 participants
@byashimov
Copy link

byashimov commented Nov 21, 2018

It seems like the recent version of multidict brakes aiohttp somehow.

Server:

from aiohttp import web

async def hello(request):
    return web.json_response(await request.json())

app = web.Application()
app.add_routes([web.post('/', hello)])
web.run_app(app)

Client:

import asyncio
import aiohttp

async def foo(times):
    data = {'foo': 1}
    async with aiohttp.ClientSession() as session:
        for x in range(times):
            resp = await session.post('http://localhost:8080', json=data)
            if not x % 100:
                print(await resp.json())

loop = asyncio.get_event_loop()
loop.run_until_complete(foo(100000))
loop.close()

Just after 100000 cycles this simple server takes 171M of ram and doesn't release it.
4.4.2 works like a charm.

python 3.6.4
aiohttp 3.4.4 but should affect other versions multidict>=4.0,<5.0

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 21, 2018

@aio-libs-bot

This comment has been minimized.

Copy link
Collaborator

aio-libs-bot commented Nov 21, 2018

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are aio-libs/aiohttp#1656 (Memory leak), aio-libs/aiohttp#271 (Memory leak in request), aio-libs/aiohttp#133 (Memory leak when doing https request), aio-libs/aiohttp#1029 (Memory leak when doing https request), and aio-libs/aiohttp#1756 (Memory leak with aiohttp.request).

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 21, 2018

@byashimov if you think that it's a regression in multidict, you should report the problem there and post an example not using aiohttp.

@byashimov

This comment has been minimized.

Copy link

byashimov commented Nov 21, 2018

@webknjaz multidict downgrade fixes the issue. Please confirm it to downgrade the package version in setup file. I'll try to reproduce it for multidict issue tracker.

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 21, 2018

Hm.. could smth like https://twitter.com/VictorStinner/status/1062282676462333952 or cprofile help?

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 21, 2018

@byashimov the latest version is also compiled with the newest Cython

@skrawcz

This comment has been minimized.

Copy link

skrawcz commented Nov 21, 2018

I can confirm, I am seeing this same memory leak using 4.5.0 of multidict. Using 4.4.2 is fine.

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 21, 2018

Can anyone rebuild that with Cython v28.5.0 to check whether it's related to Cython or to code changes?

@byashimov

This comment has been minimized.

Copy link

byashimov commented Nov 22, 2018

@webknjaz it says:

4.5.0 (2018-11-19)
Multidict views ported from Cython to C extension (#275)

According to the changelog this is the only difference between 4.4.2 and 4.5.

@asvetlov

This comment has been minimized.

Copy link
Member

asvetlov commented Nov 22, 2018

Guys, thank for the report.
I'm working on the fix.

@asvetlov

This comment has been minimized.

Copy link
Member

asvetlov commented Nov 22, 2018

Fixed by #306

@asvetlov asvetlov closed this Nov 22, 2018

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 22, 2018

@byashimov it's because I did not include dependency upgrade when doing the release
Ref #304

@delfick

This comment has been minimized.

Copy link

delfick commented Nov 26, 2018

I don't think #306 fixed the issue

@webknjaz

This comment has been minimized.

Copy link
Member

webknjaz commented Nov 26, 2018

@asvetlov keeping this open since the issue was reported about the client side.

@webknjaz webknjaz reopened this Nov 26, 2018

@asvetlov asvetlov transferred this issue from aio-libs/aiohttp Nov 26, 2018

@aio-libs-bot

This comment has been minimized.

Copy link
Collaborator

aio-libs-bot commented Nov 26, 2018

GitMate.io thinks possibly related issues are #105 (istr causes memory leak), #274 (4.4.0a1 failed ), #280 (Can't install multidict 4.4.0 in docker alpine image), #286 (4.4.1 traceback with python 3.5.2), and #62 (Multidict preserves order- document it somewhere).

@aio-libs-bot aio-libs-bot added the bug label Nov 26, 2018

asvetlov added a commit that referenced this issue Nov 28, 2018

asvetlov added a commit that referenced this issue Nov 28, 2018

@webknjaz webknjaz referenced this issue Nov 30, 2018

Open

Transferred issues ain't properly tracked #1315

1 of 2 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment