Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filter on a field for a set of values (OR between values) #137

Closed
apelliciari opened this issue Feb 16, 2014 · 38 comments
Closed

filter on a field for a set of values (OR between values) #137

apelliciari opened this issue Feb 16, 2014 · 38 comments

Comments

@apelliciari
Copy link

Hi,

i don't know if is the right place to ask, because it's not an issue, just a question.

I'm using django-filter with Django-REST-Framework to build APIs for my app (great job by the way to both of you!)

I would like to know if there's a way to get this filter working

http://example.com/api/products?category=clothing,shoes

to get all products with category "clothing" OR category "shoes".

I've tried with

http://example.com/api/products?category=clothing&category=shoes

but as expected it does an AND between the filters and retrieve only the shoes products (i suppose because it's the last filter).

@hanifvirani
Copy link

I have the exact same question.

@Flowerowl
Copy link

I have the exact same question.too

@guialante
Copy link

I have the exact question too

@Flowerowl
Copy link

i have an idea: overwrite MultipleChoiceFilter:

from django.forms.fields import MultipleChoiceField
from django_filters.filters import MultipleChoiceFilter
...
class MultipleField(MultipleChoiceField):
def valid_value(self, value):
return True

class MultipleFilter(MultipleChoiceFilter):
field_class = MultipleField

class xxxFilter(django.filters.FilterSet):
object_id = MultipleFilter(look_up='in')
class Meta(object):
model = yyy
fields = ['object_id']

@ddevlin
Copy link

ddevlin commented Mar 20, 2014

Update 2015-04-17: As @TankOs pointed out below a change in django-filter required this solution to be adjusted, I'm correcting it here in case anyone uses this code without reading further. @kmwenja also posted a nice solution below that includes sanitisation and type coercion (this solution assumes you're filtering on string fields).

@apelliciari, I have the same use case as you (DRF, need comma separated OR filter), this is my solution:

from django_filters import Filter
from django_filters.fields import Lookup

from .models import Product

class ListFilter(Filter):
    def filter(self, qs, value):
        value_list = value.split(u',')
        return super(ListFilter, self).filter(qs, Lookup(value_list, 'in'))

class ProductFilterSet(django_filters.FilterSet):
    category = ListFilter(name='categories__slug')

    class Meta:
        model = Product
        fields = ['category']

To anybody else, YMMV outside use with Django REST Framework

@xordoquy
Copy link
Contributor

xordoquy commented Jun 4, 2014

For the record, http://example.com/api/products?category=clothing&category=shoes does work provided you use MultipleChoiceFilter which does an OR between the two.

@jdotjdot
Copy link

jdotjdot commented Aug 5, 2014

The MultipleChoiceFilter hasn't even been working for me using that style.

@tiagoboldt
Copy link

+1. @ddevlin solution works perfectly, I suggest adding it to the lib.

@josh-t
Copy link

josh-t commented Sep 2, 2014

Thank you @ddevlin!

@carltongibson
Copy link
Owner

Hmmm. This is curious. I'm looking at the source code for MultipleChoiceFilter and it's definitely OR-ing — so it should work.

Anyone fancy chucking a PR together with a failing test? And if so, @ddevlin's fix in a separate commit. ;)

@carltongibson
Copy link
Owner

Can anyone paste an example of how you're setting up the filter? Basically the ?category=clothing&category=shoes with MultipleChoiceFilter should work but this issue seems too frequent (both here and on Stack Overflow) for it to be just nothing. (It may be that it's a documentation issue.)

... and retrieve only the shoes products (i suppose because it's the last filter).

This sounds like the QueryDict is having __getItem__ — or equivalent — called rather than getList somewhere. (That would explain it at any rate.) If I can see the set up then I can step through and confirm it either way.

(There's a separate issue about the ?a=1,2,3 syntax. That is on my list to come back to at a later date.)

@carltongibson
Copy link
Owner

Also — can you confirm DRF and django-filter versions?

Thanks!

@ddevlin
Copy link

ddevlin commented Oct 8, 2014

Hi @carltongibson, I don't believe this is a django-filter issue.

When DRF instantiates the FilterSet, it passes request.QUERY_PARAMS as the data argument (see this line). If the url was http://example.com/api/products?category=clothing&category=shoes then the QueryDict passed to the django-filter FilterSet won't contain the first query parameter as the second one will have overwritten it (in this example, if I put a breakpoint in the FilterSet __init__ the data argument is django.http.request.QueryDict({u'category': u'shoes'})).

DRF's request.QUERY_PARAMS is just an alias for .GET in the Django request that DRF's request is wrapping (see here) so this deduplication of query parameters happens to the request before DRF is involved. I think this makes the ?category=clothing&category=shoes style impossible to support.

@carltongibson
Copy link
Owner

But it should work like this, no...?

In [1]: from django.http.request import QueryDict

In [2]: qd = QueryDict('category=clothing&category=shoes')
---------------------------------------------------------------------------
ImproperlyConfigured                      Traceback (most recent call last)

[Hmmm. Accursed DJANGO_SETTINGS_MODULE Lets skip that shall we...]

In [3]: qd = QueryDict(u'category=clothing&category=shoes', encoding='utf8')

In [4]: qd
Out[4]: <QueryDict: {u'category': [u'clothing', u'shoes']}>

In [5]: qd["category"]
Out[5]: u'shoes'

In [6]: qd.get("category")
Out[6]: u'shoes'

In [7]: qd.getlist("category")
Out[7]: [u'clothing', u'shoes']

(This is more or less the example from the docs)

So assuming getlist is called in all the right places, it should work.

(Thanks for coming back!)

@ddevlin
Copy link

ddevlin commented Oct 8, 2014

I spoke too soon, I wasn't aware of request.GET.getlist('var') 😄

@carltongibson
Copy link
Owner

if I put a breakpoint in the FilterSet __init__ the data argument is django.http.request.QueryDict({u'category': u'shoes'})

Is that an actual, "I put a breakpoint there, and that was the value"? — If so that's wrong. We should be getting the equivalent to my:

 Out[4]: <QueryDict: {u'category': [u'clothing', u'shoes']}>

@ddevlin
Copy link

ddevlin commented Oct 8, 2014

Actual, but it appears to be a side effect of how werkzeug's debugger represents dicts, when I use IPython I get the same <QueryDict: {u'category': [u'clothing', u'shoes']}> as you do.

@BrickXu
Copy link

BrickXu commented Dec 3, 2014

Specify widget=SelectMultiple works well for my project.

@TankOs
Copy link

TankOs commented Jan 7, 2015

Just wanted to let you guys know that a change in django-filter requires an adjusted @ddevlin solution. ;-)

from django_filters import Filter
from django_filters.fields import Lookup

class ListFilter( Filter ):
  def filter( self, qs, value ):
    return super( ListFilter, self ).filter( qs, Lookup( value.split( u"," ), "in") )

@tuky
Copy link

tuky commented Feb 23, 2015

i think, most people cannot use MultipleChoiceFilter directly, because they want to OR-filter on fields that are not a MultipleChoiceField, e.g. id: http://example.com/api/products?id=2&id=3. Right now, this can only be done with the solution in #137 (comment)

@kmwenja
Copy link

kmwenja commented Mar 6, 2015

Does @ddevlin solution work for any type of filter (ie Number, Char)?

@carltongibson
Copy link
Owner

You might have to adjust types, but it should do (more or less)

@kmwenja
Copy link

kmwenja commented Mar 6, 2015

@carltongibson so does that mean inheriting from different filter types or actually typing the split values?

@carltongibson
Copy link
Owner

Yeah, you'll have to implement the filter yourself (but it's only small). split will give you a list of strings, if you're filtering on, say, an IntegerField you'll need to covert the value.

@kmwenja
Copy link

kmwenja commented Mar 7, 2015

@carltongibson you were right, here's my implementation:

class ListFilter(CharFilter):

    def sanitize(self, value_list):
        """
        remove empty items in case of ?number=1,,2
        """
        return [v for v in value_list if v != u'']

    def customize(self, value):
        return value

    def filter(self, qs, value):
        multiple_vals = value.split(u",")
        multiple_vals = self.sanitize(multiple_vals)
        multiple_vals = map(self.customize, multiple_vals)
        actual_filter = django_filters.fields.Lookup(multiple_vals, 'in')
        return super(ListFilter, self).filter(qs, actual_filter)

So for filtering integers:

class ListIntegerFilter(ListFilter):

    def customize(self, value):
        return int(value)

@zoidyzoidzoid
Copy link
Contributor

(There's a separate issue about the ?a=1,2,3 syntax. That is on my list to come back to at a later date.)

Sorry to post this on this issue, but I really can't find the issue. @carltongibson . Is it closed?

@carltongibson
Copy link
Owner

Hey @zoidbergwill — No problem — and "Not exactly".

The are two options if you want to support the 1,2,3 syntax — a custom filter(as per the examples in this thread) or (I'm coming round to thinking) a custom widget that implements value_from_datadict to return the list you're after.

(For this latter suggestion see my similar comment on #187)

If there's anything specific I'm happy for an issue on the 1,2,3 syntax to be opened — but I'm inclined to see it as a documentation issue — I'm very keen to encourage users to implement the project specific filters and widgets they need.

(I really must sit down and give the docs the love they need.)

@zoidyzoidzoid
Copy link
Contributor

I ended up using a custom simple CommaSeparatedValueFilter to get the behaviour I was looking for.

It's not a difficult filter to write, but for some reason I expected it to work out the box. I guess it is more of a documentation issue.

While reading these docs and seeing how lists worked in the queries when using the FilterSet class directly in the tests, I thought it'd work, until I looked through the code properly.

@tiagoboldt
Copy link

this is a recurring question and a requirement for many. I'm all in favor of having an implementation available in the framework.

@carltongibson
Copy link
Owner

@tiagoboldt Are you talking about the Or filtering or the 1,2,3 syntax?

The Or filtering works — this ticket is only still open because I haven't had the time to sit down and go over every corner of it to make sure there's nothing lurking.

As for supporting 1,2,3 query param syntax, I'll grant it needs documenting but a custom widget here is a simple thing.

I'm all in favor of having an implementation available in the framework.

PRs welcome.

@tiagoboldt
Copy link

@carltongibson 1,2,3 syntax. I've been using django-rest for some time and every project the need for such custom filter appears.

@visigoth
Copy link

visigoth commented Aug 2, 2015

i just ran across this issue in a project i'm working on. i used some of the above code to come up with a similar solution that works across ManyToMany relationships. that is, i have a Foo model with bars=model.ManyToManyField(Bar). when i tried the code from @ddevlin and @kmwenja , i ran into the fact that Relationship fields do not support Lookup. however, you can use Q expressions to achieve the desired filter expression. Here's my replacement code:

class ListFilter(django_filters.Filter):
  def __init__(self, filter_value=lambda x: x, **kwargs):
    super(ListFilter, self).__init__(**kwargs)
    self.filter_value_fn = filter_value

  def sanitize(self, value_list):
    return [v for v in value_list if v != u'']

  def filter(self, qs, value):
    values = value.split(u",")
    values = self.sanitize(values)
    values = map(self.filter_value_fn, values)
    f = Q()
    for v in values:
      kwargs = {self.name: v}
      f = f|Q(**kwargs)
    return qs.filter(f)

In the FilterSet, i created a ListFilter with the relationship syntax, e.g. entries = ListFilter(name='entries__id', filter_value=lambda x: int(x)). I'm not too familiar with the innards of django_filters, but i oddly did not have to add it to the fields list in order to have my filter run.

@carltongibson carltongibson removed the Bug label Sep 2, 2015
@carltongibson
Copy link
Owner

I'm going to close this as it's not a bug as it stands. We'll see if #259 makes sense for adding the 1,2,3 list syntax support as a custom widget but it should be easy enough to add in user land code if you need it in the meantime.

@andybak
Copy link

andybak commented Mar 9, 2016

I'm a little confused as to why this is closed. Maybe it's a documentation issue but MultipleChoiceFilter doesn't seem to behave as expected as Tuky is states at #137 (comment)

I spent a good hour or so on this before I found this issue thread so at the very least the docs seem a nice big warning :-)

@carltongibson
Copy link
Owner

What's the bug? Where's the failing test case demonstrating it? No one has been able to pin down the supposed issue.

@rohitkeshav
Copy link

I am sorry for asking, but what is that works now? for filtering things such as ?a=1,2,3

@tiagoboldt
Copy link

From my understanding, it's:
http://URL/something/?field=value1,value2,value3
The expected query in django would be:
something.objects.filter(Q(field=value1) | Q(field=value2) | Q(field=value3))

I totally agree with the need for a native filter as such, as it has been a need I've had in every project I've used django-rest.

@carltongibson
Copy link
Owner

Yep. ?f=1&f=2&f=3 has always worked — it just requires the right widget. The new CSV stuff adds support for ?f=1,2,3 as a single value.

Check the docs, have a play. Any issues open a new ticket.

Repository owner locked and limited conversation to collaborators Mar 30, 2016
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests