With complex queries, SearchQuerySet.exclude() filters instead of excludes

[bkonkle]

Okay, here's my SearchQuerySet query, which should exclude all old or far-future events from our search results:

sqs = sqs.exclude(
    SQ(contenttype='events.event') &
    (
        SQ(date__lte=(today - SITE_SEARCH_EVENTS_FROM)) |
        SQ(date__gte=(today + SITE_SEARCH_EVENTS_TO))
    )
)

Here is the SQ.__repr__() it generates (the sites part is applied to the query earlier on):

<SQ: AND (content__exact=old AND content__exact=97 AND sites__in=1 AND NOT ((contenttype__exact=events.event AND (date__lte=2010-01-15 OR date__gte=2010-04-22))))>

Here is the xapian.Query.get_description() that the SQ produces:

Xapian::Query(((Zold OR old) AND (Z97 OR 97) AND (ZXSITES1 OR XSITES1) AND (ZXCONTENTTYPEevents.ev OR XCONTENTTYPEevents.event) AND (VALUE_RANGE 15 00010101000000 20100115000000 OR VALUE_RANGE 15 20100422000000 99990101000000)))

Because of the 3rd AND, this filters by my exclusion instead of excluding it. It should be AND_NOT (ZXCONTENTTYPE...

If I could only change that one AND to an AND_NOT, my problems would be over for now. I've been unable to do it with my own experimentation, however.

Thank you again for any help you can provide!

[bkonkle]

Well, I guess actually it needs to be an AND_NOT with an extra set of () around the subqueries. After a week of working on nothing but this, my head is spinning. :-)

[notanumber]

Sorry it took me so long to respond to this.

I think I see what's going wrong here. The problem is the result of using SQ with exclude. I'm not sure if the issue is with Haystack itself yet, or if it's the Xapian backend. I'll need to speak with Daniel before I can be sure.

In the meantime, I'd recommend building an query that combines exclusion filters and SQ by doing something like:

 sqs.filter(~SQ(
     SQ(contenttype='events.event) & SQ(date__lte=(today - SITE_SEARCH_EVENTS_FROM) & SQ(date__gte=(today + SITE_SEARCH_EVENTS_TO))
 ))

[notanumber]

Confirmed that the problem is the exclude(SQ(...)) bit. It's definitely canceling itself out. i.e., what's happening is:

sq.exclude(*args) actually, internally, calls sq.filter(~SQ(*args)), so when you pass in a SQ as the arg, you're getting:

sq.filter(~SQ(SQ(foo))

The result; the opposite of what you want.

I think you're best bet is to use filter directly with a ~SQ. This should give you the behaviour you're expecting.

I'm going to keep this ticket open until I get feedback from Daniel, just to be absolutely certain I'm understanding this is working as designed.

[notanumber]

Resolved on IRC and git messages.