Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

CMS_PERMISSION: optionally use raw id user lookups #1481

Merged
merged 9 commits into from

4 participants

@jimr

The "view restrictions" and "page permissions" inlines on the "page"
admin change forms can cause performance problems where there are many
thousands of users being put into simple select boxes. Added a new
setting (CMS_RAW_ID_USERS) that, if set, forces the inlines on that page
to use standard Django admin "raw ID" lookups rather than select boxes.
The restriction on which users may be selected should still be honoured
by the new widget because we set the 'limit_choices_to' attribute to be
the same as the queryset on the original widget.

Using raw_id_fields in combination with limit_choices_to blows up if you
have many thousands of users. For this reason, we only apply this limit
if the number of users is relatively small (< 500, though that figure is
somewhat arbitrary). If the number of users we need to limit to is
greater than that, we use the usual input field instead unless the user
is a CMS superuser, in which case we bypass the limit. Unfortunately,
this means that non-superusers won't see any benefit from this change.

jimr added some commits
@jimr jimr CMS_PERMISSION: optionally use raw id user lookups
The "view restrictions" and "page permissions" inlines on the "page"
admin change forms can cause performance problems where there are many
thousands of users being put into simple select boxes. Added a new
setting (CMS_RAW_ID_USERS) that, if set, forces the inlines on that page
to use standard Django admin "raw ID" lookups rather than select boxes.
The restriction on which users may be selected should still be honoured
by the new widget because we set the 'limit_choices_to' attribute to be
the same as the queryset on the original widget.

Using raw_id_fields in combination with limit_choices_to blows up if you
have many thousands of users. For this reason, we only apply this limit
if the number of users is relatively small (< 500, though that figure is
somewhat arbitrary). If the number of users we need to limit to is
greater than that, we use the usual input field instead unless the user
is a CMS superuser, in which case we bypass the limit. Unfortunately,
this means that non-superusers won't see any benefit from this change.
4afe5e1
@jimr jimr Removed unused import 1ea6e55
@jimr jimr .count() rather than len() for performance boost 2b1b614
@digi604
Collaborator

Ok this needs docs and it needs merge with develop. I don't think we gonna release an other 2.2 release for this.

@jimr

OK, I'll add some docs & remove the version bump.

@jimr jimr CMS_RAW_ID_USERS: added default setting + docs
Also removed the previous version bump so it can merge cleanly upstream.
b3183f2
@digi604
Collaborator

Looks good to me... @ojii could you have a look as well?

cms/admin/forms.py
((5 lines not shown))
page = forms.ModelChoiceField(Page, label=_('user'), widget=HiddenInput(), required=True)
def __init__(self, *args, **kwargs):
super(PagePermissionInlineAdminForm, self).__init__(*args, **kwargs)
user = get_current_user() # current user from threadlocals
- self.fields['user'].queryset = get_subordinate_users(user)
- self.fields['user'].widget.user = user # assign current user
+ sub_users = get_subordinate_users(user)
+
+ limit_choices = True
+ use_raw_id = False
+ if hasattr(settings, 'CMS_RAW_ID_USERS') and settings.CMS_RAW_ID_USERS:
@ojii Collaborator
ojii added a note

if getattr(settings, 'CMS_RAW_ID_USERS', False):

@jimr
jimr added a note

Yes, that would be clearer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
cms/admin/permissionadmin.py
@@ -22,6 +22,15 @@ class PagePermissionInlineAdmin(admin.TabularInline):
classes = ['collapse', 'collapsed']
exclude = ['can_view']
+ def __getattribute__(self, name):
@ojii Collaborator
ojii added a note

is there no other way?!

@jimr
jimr added a note

It's the only way I could find of configuring raw_id_fields based on settings. If there's a better way I'll happily change it.

@ojii Collaborator
ojii added a note
@property
def  raw_id_fields(self):
    if ....
@jimr
jimr added a note

Unfortunately this isn't possible because of the way Django validates the form (it checks to see whether the raw_id_fields is an instance of list or tuple which would fail if it's a property).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ojii
Collaborator

I have a few isuses with the patch itself, but on top of that I'm (as we all know by now) not a huge fan of adding new settings.

If this is such a good thing, why don't we just make it the default?

@digi604
Collaborator

Good point @ojii. Remove the setting and make it an Raw ID field if there are more 500 users? Maybe even 200?

@jimr

I only made it a setting because I didn't want to alter default behaviour. With thousands of user accounts, it's currently infeasible to use CMS_PERMISSION because of the issues addressed by this patch, so making it default would make sense to me.

@FrankBie

Would it not be better to rename the setting from CMS_RAW_ID_USERS to CMS_RAW_ID_USERS_LIMIT_COUNT defaulting to None and otherwise taking a configurable limit amount?

Pro: you can put it to the limit you like without agreeing to the 500.

@digi604
Collaborator

please remerge with develop and i think too that the configurable limit would be better...

cms/conf/global_settings.py
@@ -26,6 +26,9 @@
# Whether to enable permissions.
CMS_PERMISSION = False
+# Whether to use raw ID lookups for users when CMS_PERMISSION is set to True
+CMS_RAW_ID_USERS = False
@ojii Collaborator
ojii added a note

with regards to the limit, how about making this setting either False (disable the raw ID lookup) or an integer which is the limit?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jimr

OK, sounds good. I'll bring my branch up to date & make the limit configurable.

@jimr

@ojii should I put CMS_RAW_ID_USERS in DEFAULTS in cms/utils/conf.py as RAW_ID_USERS? Seems the arrangement of settings has changed rather a lot since I started.

@ojii
Collaborator

yes please

jimr added some commits
@jimr jimr Merge branch 'develop' into raw_id_users
Conflicts:
	cms/conf/global_settings.py
2481bf3
@jimr jimr Only use raw ID above a certain number of users
So the RAW_ID_USERS setting should now be either left as False or set to
a positive integer which will be the threshold for the number of users
above which the raw ID widget will be used.
ec9f8d8
@jimr jimr import typo 61055b9
cms/admin/permissionadmin.py
@@ -27,6 +27,16 @@ class PagePermissionInlineAdmin(TabularInline):
exclude = ['can_view']
extra = 0 # edit page load time boost
+ def __getattribute__(self, name):
@ojii Collaborator
ojii added a note

github seems to have lost this discussion, but this works for me:

In [1]: class A:
   ...:     @property
   ...:     def b(self):
   ...:         return []
   ...:     

In [2]: a = A()

In [3]: isinstance(a.b, list)
Out[3]: True

In [4]: isinstance(a.b, (list, tuple))
Out[4]: True
@jimr
jimr added a note

Django checks it on the class, not an instance, otherwise it would indeed work.

https://github.com/django/django/blob/master/django/contrib/admin/validation.py#L267

@ojii Collaborator
ojii added a note
class classproperty(object):

    def __init__(self, fget):
        self.fget = fget

    def __get__(self, owner_self, owner_cls):
        return self.fget(owner_cls)

## -- End pasted text --

In [2]: class A:
   ...:     @classproperty
   ...:     def b(cls):
   ...:         return []
   ...:     

In [3]: A.b
Out[3]: []
@jimr
jimr added a note

OK, so that would work, but do you think it's better than just using __getattribute__? Up to you, but IMO the original code is easier to understand.

@ojii Collaborator
ojii added a note

arguably, but I just really am not a fan of __getattribute__. had too many issues with it in the past...

@jimr
jimr added a note

Fair enough. Let me know if this is blocking the merge and I'll change it, but my personal preference would be to leave as-is.

@ojii Collaborator
ojii added a note

had a chat with @digi604 and @stefanfoulis and we all three prefer classproperty over __getattribute__.

I think the best place to put that helper class is probably cms.utils.helpers, so if you could change this bit to make use of that technique, that would be highly appreciated.

@jimr
jimr added a note

Okie doke.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ojii ojii commented on the diff
cms/admin/forms.py
((5 lines not shown))
page = forms.ModelChoiceField(Page, label=_('user'), widget=HiddenInput(), required=True)
def __init__(self, *args, **kwargs):
super(PagePermissionInlineAdminForm, self).__init__(*args, **kwargs)
user = get_current_user() # current user from threadlocals
- self.fields['user'].queryset = get_subordinate_users(user)
- self.fields['user'].widget.user = user # assign current user
+ sub_users = get_subordinate_users(user)
+
+ limit_choices = True
+ use_raw_id = False
+
+ # Unfortunately, if there are > 500 users in the system, non-superusers
+ # won't see any benefit here because if we ask Django to put all the
+ # user PKs in limit_choices_to in the query string of the popup we're
+ # in danger of causing 414 errors so we fall back to the normal input
+ # widget.
+ if get_cms_setting('RAW_ID_USERS'):
+ if sub_users.count() < 500:
@ojii Collaborator
ojii added a note

isn't this where it should use the variable setting?

@jimr
jimr added a note

The threshold check to decide whether we should try to use a raw ID field is applied in cms/admin/permissionadmin.py. The "< 500" check applied here is only to prevent the raw ID blowing up with a 414 error, and there's little point making this configurable, IMO (though 500 is a little arbitrary).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jimr jimr Switch from __getattribute__ to classproperty
Added the new classproperty helper in cms.utils.helpers.
9a778f5
@digi604 digi604 merged commit bd166e3 into from
@jimr

Thanks!

@jimr jimr deleted the branch
@digi604
Collaborator

Thank YOU!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Sep 14, 2012
  1. @jimr

    CMS_PERMISSION: optionally use raw id user lookups

    jimr authored
    The "view restrictions" and "page permissions" inlines on the "page"
    admin change forms can cause performance problems where there are many
    thousands of users being put into simple select boxes. Added a new
    setting (CMS_RAW_ID_USERS) that, if set, forces the inlines on that page
    to use standard Django admin "raw ID" lookups rather than select boxes.
    The restriction on which users may be selected should still be honoured
    by the new widget because we set the 'limit_choices_to' attribute to be
    the same as the queryset on the original widget.
    
    Using raw_id_fields in combination with limit_choices_to blows up if you
    have many thousands of users. For this reason, we only apply this limit
    if the number of users is relatively small (< 500, though that figure is
    somewhat arbitrary). If the number of users we need to limit to is
    greater than that, we use the usual input field instead unless the user
    is a CMS superuser, in which case we bypass the limit. Unfortunately,
    this means that non-superusers won't see any benefit from this change.
  2. @jimr

    Removed unused import

    jimr authored
Commits on Oct 16, 2012
  1. @jimr
Commits on Oct 18, 2012
  1. @jimr

    CMS_RAW_ID_USERS: added default setting + docs

    jimr authored
    Also removed the previous version bump so it can merge cleanly upstream.
Commits on Nov 12, 2012
  1. @jimr

    clearer conditional check

    jimr authored
Commits on Apr 5, 2013
  1. @jimr

    Merge branch 'develop' into raw_id_users

    jimr authored
    Conflicts:
    	cms/conf/global_settings.py
  2. @jimr

    Only use raw ID above a certain number of users

    jimr authored
    So the RAW_ID_USERS setting should now be either left as False or set to
    a positive integer which will be the threshold for the number of users
    above which the raw ID widget will be used.
  3. @jimr

    import typo

    jimr authored
Commits on Apr 8, 2013
  1. @jimr

    Switch from __getattribute__ to classproperty

    jimr authored
    Added the new classproperty helper in cms.utils.helpers.
This page is out of date. Refresh to see the latest.
View
47 cms/admin/forms.py
@@ -10,7 +10,7 @@
from cms.utils.page import is_valid_page_slug
from cms.utils.page_resolver import get_page_from_path, is_valid_url
from cms.utils.permissions import (get_current_user, get_subordinate_users,
- get_subordinate_groups)
+ get_subordinate_groups, get_user_permission_level)
from cms.utils.urlutils import any_path_re
from django import forms
from django.contrib.auth.forms import UserCreationForm
@@ -196,14 +196,53 @@ class PagePermissionInlineAdminForm(forms.ModelForm):
level or under him in choosen page tree, and users which were created by him,
but aren't assigned to higher page level than current user.
"""
- user = forms.ModelChoiceField('user', label=_('user'), widget=UserSelectAdminWidget, required=False)
page = forms.ModelChoiceField(Page, label=_('user'), widget=HiddenInput(), required=True)
def __init__(self, *args, **kwargs):
super(PagePermissionInlineAdminForm, self).__init__(*args, **kwargs)
user = get_current_user() # current user from threadlocals
- self.fields['user'].queryset = get_subordinate_users(user)
- self.fields['user'].widget.user = user # assign current user
+ sub_users = get_subordinate_users(user)
+
+ limit_choices = True
+ use_raw_id = False
+
+ # Unfortunately, if there are > 500 users in the system, non-superusers
+ # won't see any benefit here because if we ask Django to put all the
+ # user PKs in limit_choices_to in the query string of the popup we're
+ # in danger of causing 414 errors so we fall back to the normal input
+ # widget.
+ if get_cms_setting('RAW_ID_USERS'):
+ if sub_users.count() < 500:
@ojii Collaborator
ojii added a note

isn't this where it should use the variable setting?

@jimr
jimr added a note

The threshold check to decide whether we should try to use a raw ID field is applied in cms/admin/permissionadmin.py. The "< 500" check applied here is only to prevent the raw ID blowing up with a 414 error, and there's little point making this configurable, IMO (though 500 is a little arbitrary).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ # If there aren't too many users, proceed as normal and use a
+ # raw id field with limit_choices_to
+ limit_choices = True
+ use_raw_id = True
+ elif get_user_permission_level(user) == 0:
+ # If there are enough choices to possibly cause a 414 request
+ # URI too large error, we only proceed with the raw id field if
+ # the user is a superuser & thus can legitimately circumvent
+ # the limit_choices_to condition.
+ limit_choices = False
+ use_raw_id = True
+
+ # We don't use the fancy custom widget if the admin form wants to use a
+ # raw id field for the user
+ if use_raw_id:
+ from django.contrib.admin.widgets import ForeignKeyRawIdWidget
+ # This check will be False if the number of users in the system
+ # is less than the threshold set by the RAW_ID_USERS setting.
+ if isinstance(self.fields['user'].widget, ForeignKeyRawIdWidget):
+ # We can't set a queryset on a raw id lookup, but we can use
+ # the fact that it respects the limit_choices_to parameter.
+ if limit_choices:
+ self.fields['user'].widget.rel.limit_choices_to = dict(
+ id__in=list(sub_users.values_list('pk', flat=True))
+ )
+ else:
+ self.fields['user'].widget = UserSelectAdminWidget()
+ self.fields['user'].queryset = sub_users
+ self.fields['user'].widget.user = user # assign current user
+
self.fields['group'].queryset = get_subordinate_groups(user)
def clean(self):
View
13 cms/admin/permissionadmin.py
@@ -4,10 +4,11 @@
from cms.exceptions import NoPermissionsException
from cms.models import Page, PagePermission, GlobalPagePermission, PageUser
from cms.utils.conf import get_cms_setting
+from cms.utils.helpers import classproperty
from cms.utils.permissions import get_user_permission_level
from copy import deepcopy
-from django.conf import settings
from django.contrib import admin
+from django.contrib.auth.models import User
from django.template.defaultfilters import title
from django.utils.translation import ugettext as _
@@ -26,7 +27,15 @@ class PagePermissionInlineAdmin(TabularInline):
classes = ['collapse', 'collapsed']
exclude = ['can_view']
extra = 0 # edit page load time boost
-
+
+ @classproperty
+ def raw_id_fields(cls):
+ # Dynamically set raw_id_fields based on settings
+ threshold = get_cms_setting('RAW_ID_USERS')
+ if threshold and User.objects.count() > threshold:
+ return ['user']
+ return []
+
def queryset(self, request):
"""
Queryset change, so user with global change permissions can see
View
2  cms/utils/conf.py
@@ -32,6 +32,8 @@ def wrapper():
'TEMPLATE_INHERITANCE': True,
'PLACEHOLDER_CONF': {},
'PERMISSION': False,
+ # Whether to use raw ID lookups for users when PERMISSION is True
+ 'RAW_ID_USERS': False,
'PUBLIC_FOR': 'all',
'CONTENT_CACHE_DURATION': 60,
'SHOW_START_DATE': False,
View
33 cms/utils/helpers.py
@@ -66,4 +66,35 @@ def make_revision_with_plugins(obj, user=None, message=None):
revision_context.add_to_context(revision_manager, plugin, bpadapter.get_version_data(plugin, VERSION_CHANGE))
def find_placeholder_relation(obj):
- return 'page'
+ return 'page'
+
+
+class classproperty(object):
+ """Like @property, but for classes, not just instances.
+
+ Example usage:
+
+ >>> from cms.utils.helpers import classproperty
+ >>> class A(object):
+ ... @classproperty
+ ... def x(cls):
+ ... return 'x'
+ ... @property
+ ... def y(self):
+ ... return 'y'
+ ...
+ >>> A.x
+ 'x'
+ >>> A().x
+ 'x'
+ >>> A.y
+ <property object at 0x2939628>
+ >>> A().y
+ 'y'
+
+ """
+ def __init__(self, fget):
+ self.fget = fget
+
+ def __get__(self, owner_self, owner_cls):
+ return self.fget(owner_cls)
View
27 docs/getting_started/configuration.rst
@@ -530,6 +530,33 @@ a certain page all users he creates can, in turn, only edit this page. Naturally
he can limit the rights of the users he creates even further, allowing them to see
only a subset of the pages to which he is allowed access.
+.. setting:: CMS_RAW_ID_USERS
+
+CMS_RAW_ID_USERS
+================
+
+Default: ``False``
+
+This setting only applies if :setting:`CMS_PERMISSION` is ``True``
+
+The "view restrictions" and "page permissions" inlines on the
+:class:`cms.models.Page` admin change forms can cause performance problems
+where there are many thousands of users being put into simple select boxes. If
+set to a positive integer, this setting forces the inlines on that page to use
+standard Django admin raw ID widgets rather than select boxes if the number of
+users in the system is greater than that number, dramatically improving
+performance.
+
+.. note:: Using raw ID fields in combination with ``limit_choices_to`` causes
+ errors due to excessively long URLs if you have many thousands of
+ users (the PKs are all included in the URL of the popup window). For
+ this reason, we only apply this limit if the number of users is
+ relatively small (fewer than 500). If the number of users we need to
+ limit to is greater than that, we use the usual input field instead
+ unless the user is a CMS superuser, in which case we bypass the
+ limit. Unfortunately, this means that non-superusers won't see any
+ benefit from this setting.
+
.. setting:: CMS_PUBLIC_FOR
CMS_PUBLIC_FOR
Something went wrong with that request. Please try again.