Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve relation choices limiting #3329

Closed
rubendura opened this issue Aug 25, 2015 · 4 comments
Closed

Improve relation choices limiting #3329

rubendura opened this issue Aug 25, 2015 · 4 comments
Labels
Milestone

Comments

@rubendura
Copy link
Contributor

rubendura commented Aug 25, 2015

Building upon #3313 comments. This is a follow up for #3309.

There can be some memory issues with RelatedField.choices when dealing with really big querysets (~400k items in my case). This property fetches all models in the queryset and then builds a massive OrderedDict with them, which basically can clog the memory of many systems.

I suggest that we slice the queryset using html_cutoff before evaluating the queryset and building the returned OrderedDict. It seems to do the trick for me, but it'd be better if someone can confirm if this is a valid solution.

This is the code snippet that I used to do a quick (and so far successful) test. I've only added the bit slicing the queryset.

In relations.py:144 (RelatedField.choices)

@property
def choices(self):
    queryset = self.get_queryset()
    if queryset is None:
        # Ensure that field.choices returns something sensible
        # even when accessed with a read-only field.
        return {}

    return OrderedDict([
        (
            six.text_type(self.to_representation(item)),
            self.display_value(item)
        )
        for item in queryset[:self.html_cutoff]
    ])
@blag
Copy link
Contributor

blag commented Jan 27, 2016

According to the docs, html_cutoff can be None, so I think the slicing needs to be done in a conditional. That's really all my PR #3877 does differently.

@tomchristie tomchristie added this to the 3.3.3 Release milestone Jan 27, 2016
@tomchristie
Copy link
Member

tomchristie commented Jan 27, 2016

Milestoning each of this, #3877 and #3330 for review.

@xordoquy xordoquy modified the milestones: 3.3.3 Release, 3.3.4 Release, 3.4.0 Release Feb 11, 2016
@wimglenn
Copy link
Contributor

wimglenn commented Feb 22, 2016

@blag slicing like [:None] is OK though, isn't it?

@blag
Copy link
Contributor

blag commented Feb 22, 2016

@wimglenn: I didn't know this, but apparently yes it is. Thanks for pointing this out! 😄

I'll try to update my PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants