No field filters form on the browsable API

[borfast]

I have a rest_framework_filters.FilterSet-derived class which works perfectly for filtering but it doesn't show any field filters in the browsable API.

I've been scouring the web for information about this but just can't find anything. Is it a known limitation that I somehow missed, or is it a bug? Maybe related to #40 (Use new DRF-friendly FilterSet & Backend)?

[rpkilby]

Hi @borfast. drf-filters should be compatible w/ drf 3.3 but targets 3.2, as that's what was out at the time. The browsable API filters were added as part of 3.3, so I'm unsure if they are compatible. I looked into the source, and there's nothing that indicates to me that the form would not exist/be empty.

Relevant code is in the filter backend, which is called by the renderer. I would start by prodding there.

[borfast]

This is getting weirder and weirder...

I loaded the code in the debugger and followed the flow of execution all the way to the entrails of Django's templates system. In NodeList.render() I could watch the various HTML bits and pieces being assembled and I was still seeing my filter class in the context that is passed to this method.

Whether I derive my filter class from rest_framework_filters.FilterSet or from rest_framework.filters.FilterSet, I see the same context being passed to NodeList.render() and an instance of crispy_forms.templatetags.crispy_forms_tags.CrispyFormNode always gets added to the template node list to be rendered.

Following the code further, everything that gets passed from here on seems to be the same for both parent-class scenarios but obviously something must be amiss. I just can't figure out what it is.

[rpkilby]

If you manually instantiate your FilterSet and call to_html with the appropriate args, do you get meaningful output? At that point, it should be a fully rendered string of HTML.

Also, I know there is special handling for when you have crispy forms & django-filter installed. Try uninstalling crispy forms and see if the form at least shows up.

[borfast]

Uninstalling Crispy Forms doesn't seem to alter anything, the form is still missing.

As for instantiating my FilterSet and manually calling to_html() on it, I'm not sure what parameters I should pass to it.

But I did some more digging and I think I'm zooming in on the problem. This time I followed the code into crispy_forms/templatetags/crispy_forms_tags.py and realised that helper.template_pack and helper.layout are both set when using rest_framework.filters.FilterSet but not when using rest_framework_filters.FilterSet.

More specifically, helper is set to None on line 105 and later, on line 113 it is set to something else, but that something else comes from actual_form.helper, and actual_form is set on line 107 with actual_form = form.resolve(context).

I'm sure that if I keep pulling the string, I will eventually go back to line 105 and find out that form is missing something, which will lead me to look at context.render_context[self] and the stuff that gets added there back in line 101.

I'll try to dig a little deeper and see what I can find.

[rpkilby]

to_html is defined here. Looks like you need:

• a request object, although you could mock this out with a fake request
• the queryset for the model
• the relevant viewset

[borfast]

I had already tried that - sorry, forgot to mention it on my previous comment.
But no joy, the resulting HTML code I get is still just an empty form.

Here's something I should probably have done sooner - my code (all crammed into a single place for simplicity):

from rest_framework import viewsets, permissions, filters
import rest_framework_filters as rff
from .models import Reading

class ReadingViewSet(viewsets.ModelViewSet):
    """
    Returns a list of readings.
    """
    queryset = Reading.objects.all()
    serializer_class = ReadingSerializer
    permission_classes = (permissions.IsAuthenticated, ReadingBelongsToSameClient)
    filter_backends = (ReadingFilterBackend, rff.backends.DjangoFilterBackend)
    filter_class = ReadingFilter

    def get_queryset(self):
        """ Just for testing purposes."""
        dfb = DjangoFilterBackend()
        qs = Reading.objects.all()[:10]
        html = dfb.to_html(self.request, qs, self)

        return super(ReadingViewSet, self).get_queryset()


class ReadingSerializer(serializers.ModelSerializer):
    class Meta:
        model = Reading
        fields = ('id', 'value', 'meter', 'datetime', 'consumption', 'cost')


class ReadingBelongsToSameClient(permissions.BasePermission):
    """ Custom API permissions for accessing readings. """

    def has_object_permission(self, request, view, obj):
        # Only allow a user who belongs to the same client as the object
        return obj.client == request.user.userprofile.client


class ReadingFilterBackend(filters.BaseFilterBackend):
    """
    Filter that only allows users to see their own objects.
    """
    def filter_queryset(self, request, queryset, view):
        return queryset.filter(client=request.user.userprofile.client)


class ReadingFilter(rff.FilterSet):
    start_date = rff.DateTimeFilter(name='datetime', lookup_type='gte')
    end_date = rff.DateTimeFilter(name='datetime', lookup_type='lte')

    class Meta:
        model = Reading
        fields = ['price', 'channel', 'unit', 'meter', 'resource_type']

[rpkilby]

Okay - so I finally figured what's going on. The following works:

f = ReadingFilter({}, Reading.objects.all()).form

So why does the template render an empty form? Because reasons. Specifically,

Technically, when the template system encounters a dot, it tries the following lookups, in this order:

• Dictionary lookup
• Attribute or method lookup
• Numeric index lookup
  If the resulting value is callable, it is called with no arguments. The result of the call becomes the template value.

This lookup order can cause some unexpected behavior with objects that override dictionary lookup.

The last sentence is particularly interesting, as that's exactly what django_filters.FilterSet does. The template language attempts to perform a dictionary lookup on 'form' first, which in turn forces .qs to evaluate. This will fail and then use normal attribute access to get the form, however...

We override .qs for performance reasons. More details are outlined in #38, but the gist of it is that related filtersets have a potentially significant performance hit. Given a moderately complex relationship graph, it's not inconceivable to have a FilterSet that has hundreds or thousands of filters. It's expensive to deep copy all of these filters (can easily double the request time), so we generate a subclass with the subset of filters that are requested. In this case, the subset is none, as there were no requested filters. This is why the form is empty.

[rpkilby]

In the short term - it's not a great solution - but I would subclass our filter backend and override to_html. Something like...

def to_html(self, request, queryset, view):
    filter_class = self.get_filter_class(view, queryset)
    if not filter_class:
        return None
    filter_instance = filter_class(request.query_params, queryset=queryset)

    # forces `form` evaluation before `qs` is called. This prevents an empty form from being cached.
    filter_instance.form

    context = {
        'filter': filter_instance
    }
    template = loader.get_template(self.template)
    return template_render(template, context)

[borfast]

Great detective job, @rpkilby! :)

I have been a bit away from this in the past couple of days because other things came up that I had to deal with but thanks a lot for looking into this and getting to the bottom of it.

Are you guys going to implement that solution in DRF-Filters or will you try to come up with something less "hacky"?

As a temporary solution I derived my ReadingFilter class from the regular rest_framework.filters.FilterSet class instead of rest_framework_filters.FilterSet but kept the start_date and end_date properties as fields from rest_framework_filters. It seems to be working just fine and I understand I won't get the benefits of the changes from DRF-Filters but could it have any unexpected side-effects from mixing the regular DRF FilterSet with the DRF-Filters fields?

[rpkilby]

I am currently attempting to restructure some of the related-filtering code. This should allow for a less hacky way of integrating with the html form.

[rpkilby]

So I ended up refactoring get_filters, and it turns out that this is the only reasonable way to fix form rendering. There are more complicated ways of disabling the get_filters call, but it's simpler to just force form evaluation before the queryset is evaluated.