idna package increases memory usage by ~20MB

[DanielGibbsNZ]

I am running an Ubuntu 16.04.1 LTS machine with Python 3.5.2 which has about 40 Python scripts loaded, each one of which uses Requests. I recently updated Requests and suddenly found I was running out of memory.

Upon further investigation I discovered that as of version 2.12.0, the idna package has been included in requests and this has caused the memory consumption for each script to increase by about 20MB.

This can be observed with the following steps:

1. Install Requests version 2.11.1
2. Run python3 -c "import requests; import time; time.sleep(60)" &.
3. While the previous command is running, run top or htop and observe the resident memory usage of the command.
4. Install Requests version 2.12.0.
5. Repeat steps 2 and 3.

Is there any way that Requests could lazily load the idna library so that this extra 20MB of memory is only used when actually working with IDNA domains?

[Lukasa]

Yeah, requests could absolutely lazily import idna. Anyone who would like to can work on a patch to defer this import. =)

[DanielGibbsNZ]

Just had a look at where idna is used (https://github.com/kennethreitz/requests/blob/7d2dfa86841fae49fa82f4f59098d5be862d1ba0/requests/models.py#L371) and it seems it's being used for every request; is there any check we can do to see whether idna needs to be loaded here?

[Lukasa]

@DanielGibbsNZ Not enormously effectively, no. In principle the check in the except block could be used, I suppose.

[DanielGibbsNZ]

Something like this perhaps?

if not unicode_is_ascii(host):
    from .packages import idna
    try:
        host = idna.encode(host, uts46=True).decode('utf-8')
    except (UnicodeError, idna.IDNAError):
        if not unicode_is_ascii(host) or host.startswith(u'*'):
            raise InvalidURL('URL has an invalid label.')

[Lukasa]

That's sensible enough, though we would want to avoid doing the unicode_is_ascii check twice.

[DanielGibbsNZ]

Hmm, looks slightly more complicated than I thought to actually get this working as I'm not sure how to get packages/__init__.py to lazily load idna. Additionally, urllib3's pyopenssl uses idna as well. I might leave this for someone more experienced than I.

[moin18]

@Lukasa: idna is also imported in urllib3/contrib/pyopenssl.py. Here: https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/contrib/pyopenssl.py#L46

In order to remove the reference of idna via garbage collector, corresponding changes in urllib3 are also required. The idea is to do lazy loading of idna package only on demand basis.

Let me know if I am on correct path. If yes, I will raise the bug and pull request to change this behavior in urllib3 as well.

Sample diff is available here: https://github.com/kennethreitz/requests/pull/3787

[Lukasa]

For the urllib3 case, we could also lazily load idna, but let's fix it one step at a time.

Looks like it's already fixed there.