Repository navigation
Improve relation choices limiting #3329
Description
Building upon #3313 comments. This is a follow up for #3309.
There can be some memory issues with RelatedField.choices when dealing with really big querysets (~400k items in my case). This property fetches all models in the queryset and then builds a massive OrderedDict with them, which basically can clog the memory of many systems.
I suggest that we slice the queryset using html_cutoff before evaluating the queryset and building the returned OrderedDict. It seems to do the trick for me, but it'd be better if someone can confirm if this is a valid solution.
This is the code snippet that I used to do a quick (and so far successful) test. I've only added the bit slicing the queryset.
In relations.py:144 (RelatedField.choices)
@property
def choices(self):
queryset = self.get_queryset()
if queryset is None:
# Ensure that field.choices returns something sensible
# even when accessed with a read-only field.
return {}
return OrderedDict([
(
six.text_type(self.to_representation(item)),
self.display_value(item)
)
for item in queryset[:self.html_cutoff]
])
Activity
blag commentedon Jan 27, 2016
According to the docs,
html_cutoffcan beNone, so I think the slicing needs to be done in a conditional. That's really all my PR #3877 does differently.lovelydinosaur commentedon Jan 27, 2016
Milestoning each of this, #3877 and #3330 for review.
wimglenn commentedon Feb 22, 2016
@blag slicing like [:None] is OK though, isn't it?
blag commentedon Feb 22, 2016
@wimglenn: I didn't know this, but apparently yes it is. Thanks for pointing this out! 😄
I'll try to update my PR.
1 remaining item