Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve relation choices limiting #3329

rubendura opened this issue Aug 25, 2015 · 4 comments

Improve relation choices limiting #3329

rubendura opened this issue Aug 25, 2015 · 4 comments


Copy link

rubendura commented Aug 25, 2015

Building upon #3313 comments. This is a follow up for #3309.

There can be some memory issues with RelatedField.choices when dealing with really big querysets (~400k items in my case). This property fetches all models in the queryset and then builds a massive OrderedDict with them, which basically can clog the memory of many systems.

I suggest that we slice the queryset using html_cutoff before evaluating the queryset and building the returned OrderedDict. It seems to do the trick for me, but it'd be better if someone can confirm if this is a valid solution.

This is the code snippet that I used to do a quick (and so far successful) test. I've only added the bit slicing the queryset.

In (RelatedField.choices)

def choices(self):
    queryset = self.get_queryset()
    if queryset is None:
        # Ensure that field.choices returns something sensible
        # even when accessed with a read-only field.
        return {}

    return OrderedDict([
        for item in queryset[:self.html_cutoff]
Copy link

blag commented Jan 27, 2016

According to the docs, html_cutoff can be None, so I think the slicing needs to be done in a conditional. That's really all my PR #3877 does differently.

@tomchristie tomchristie added this to the 3.3.3 Release milestone Jan 27, 2016
Copy link

tomchristie commented Jan 27, 2016

Milestoning each of this, #3877 and #3330 for review.

@xordoquy xordoquy modified the milestones: 3.3.3 Release, 3.3.4 Release, 3.4.0 Release Feb 11, 2016
Copy link

wimglenn commented Feb 22, 2016

@blag slicing like [:None] is OK though, isn't it?

Copy link

blag commented Feb 22, 2016

@wimglenn: I didn't know this, but apparently yes it is. Thanks for pointing this out! 😄

I'll try to update my PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

No branches or pull requests

5 participants