Requests memory leak: Caused computer to crash

[Kodingu]

So as I was working on coding a notifier for a website, I came across an issue where my computer would gradually slow down after a long period of time. I thought it was nothing until my computer crashed. I then decided to look up on this issue by testing every single line of code in my while loop until i finally found out that it was caused by the requests module.

I decided to look up on google if anyone else has gotten an issue like this and I actually did manage to find a page with someone who had this exact same problem.

My code that I used to create this memory leak was this:

import requests

ListedItems = {}

while True:
	r = requests.get("urlToGet").json()
	time.sleep(1)
	for itm in r:
		print(itm)
	del r

I even tried to delete the r variable after I got everything I needed, but it still resulted in my computer to crash after a few hours. Please note that I need to have this program running for days.

[nateprewitt]

Hi @Kodingu, we have another issue open (#4601) for something similar, but haven't been given enough information to identify a problem.

Could you please provide the information originally requested in the template used to create this ticket? It also looks like you may have been storing items in ListedItems at some point since you've included it in your repro. To verify, this is the exact code you've used to repro and you aren't storing anything from these responses?

Would you also mind testing this with the minimum required amount of code to reproduce?

e.g. Does this repro the problem for you?

import requests
import time

while True:
    r = requests.get("urlToGet")
    time.sleep(1)
    print(r.content)
    del r

[initbar]

I've used @nateprewitt's sample code and the memory usage seems stable to me:

I'm also curious whether ListItems dict was modified to keep historic or memo-istic records of the json elements.

If you do want to use a dictionary, nonetheless, and would like to have an upper bound for the contained elements, I would recommend using the following dict implementation:

from collections import OrderedDict

class limited_dict(OrderedDict):

    def __init__(self, *a, **kw):
        self.limit = kw.pop("maxsize", None)
        OrderedDict.__init__(self, *a, **kw)

    def __setitem__(self, key, value):
        OrderedDict.__setitem__(self, key, value)
        self.__check()

    def __check(self):
        if self.limit is not None:
            while len(self) > self.limit:
                self.popitem(last=False)

# test
# a = limited_dict(maxsize=10)
# for i in range(20):
#     a[str(i)] = i
#
# print a
# limited_dict([('10', 10),
#               ('11', 11),
#               ('12', 12),
#               ('13', 13),
#               ('14', 14),
#               ('15', 15),
#               ('16', 16),
#               ('17', 17),
#               ('18', 18),
#               ('19', 19)])

[nateprewitt]

Thanks so much for taking the time to look into this @initbar! It does look like there may have been more than just Requests code involved in the initial POC with the potential for lingering pointers.

I think we may need a longer look at memory usage to really rule out a leak, but that should probably be tracked in the main issue. I'm closing this out since @Kodingu hasn't provided any more information, and we'll move forward in #4601.