Performance Issue in filter_perms_queryset_by_objects

[cmaggiulli]

We are using a forked version of Guardian 2.4.0 that lives in a private artifact repository. We forked Guardian because of two separate performance issues. The issue I am addressing in this Git Issue is related to the filter_perms_queryset_by_objects function. Some information about our application:

Database Version

select aurora_version(), @@aurora_version, version();
'3.02.2', '3.02.2', '8.0.23

Django Version

Django==3.0.14

Django-Guardian Version

`forked-django-guardian==2.4.1 ( this is an internally hosted fork of 2.4.1 )

We are using Direct Foreign Keys for all User and Group Object Permissions

DRF Versions

djangorestframework==3.11.0
djangorestframework-jsonapi==3.0.0

The application has four tables that may be relevant to this issue:

1. FirstTable ( roughly 2500000 records )
2. FirstTableType ( 50 records, its a type of FirstTable )
3. FirstTableTableType ( a m2m table relating FirstTable and FirstTableType, roughly 2600000 records )
4. FirstTableUserObjectPermission ( roughly 50000000 records )

After upgrading from 2.3.0 (fork) to 2.4.0 (fork) we noticed very poor performance with PATCH requests to FirstTable . The request went from taking roughly 2s to well over 30s. We traced the offending query to line 848 in shortcuts.py
https://github.com/django-guardian/django-guardian/blob/devel/guardian/shortcuts.py#L848

It appears that this is doing a full table scan. For users with global group permissions to view FirstTable ( OOTB Django perms, not guardian object level perms ) this query runs in <1s. However, for users with only object level perms ( group or user ) the following query takes >30s

            **{'{}__in'.format(field): list(objects.values_list(
                'pk', flat=True).distinct().order_by())})

The following change seems to drastically increase performance

            **{'{}__in'.format(field): objects.values_list(
                'pk', flat=True).distinct().order_by()})

We are going to issue a MR to change this