Extend Index.index_queryset / read_queryset to see the active backend #534

Closed
wants to merge 9 commits into from

4 participants

@acdha

This allows the index to customize the queryset based on the active backend. This is extremely handy for multi-lingual work as it means the index_queryset can do things like exclude documents which aren't in the correct language, as demonstrated in the attached documentation update.

@mvdwaeter

Im using his fork right now. It is exactly what Im looking for to use multi-language indexation.

@acdha

I've rebased my branch against the current master

@acdha

This leaves one design change: currently update_index / clear_index / rebuild_index use an assumed default via --using. With this change, it would become more useful to have something like a --using=ALL or changing the implied default so the lack of --using means that all registered backends will be updated.

@toastdriven

@acdha I'd introduce a new constant for all backends & use that. If you can get tests passing, I'd say merge away.

@acdha

Cool - I'll start working on adding a new constant and getting some proper test coverage

@acdha

As of now, the tests run. The multipleindex tests were failing due to the way the Whoosh index wasn't properly cleaned up.

I still need to write some tests for the coverage path using the provided using value

@acdha

@toastdriven: I still need the corresponding update to clear_index but this works well in my testing and makes it trivial to update only a selected set of backends

@acdha

I've added tests to confirm that the management commands behave as expected. I believe this branch is now ready to merge.

@acdha

rebased against master. @toastdriven - do you want to review this? The tests pass for me locally against Solr 3.6, 4.0, ElasticSearch 0.20.1 (modulo expected failures) and Whoosh

acdha added some commits Mar 11, 2012
@acdha acdha Index index_queryset method can see the target backend
This allows backend-specific logic - for example, with language-specific
backends an index can do something like

    def index_queryset(self, using):
        return qs.filter(language=using)
6c4ccb0
@acdha acdha Make using optional for both index_queryset and read_queryset e3b35fd
@acdha

update_index.py has a few unintentional merge regressions - mostly minor (comments, etc) but I need to fix them

@toastdriven

Sorry it took so long to get back here. Looks super-solid. Once the regressions are fixed, no further review needed, :shipit:

acdha added some commits Mar 12, 2012
@acdha acdha Added multi-index examples for backend-based content filtering
This demonstrates how to filter index update / read querysets when
necessary and two examples for how to perform this filtering
77519e8
@acdha acdha Index.update: pass using through to index_queryset() call a0a6254
@acdha acdha Basic tests for filtering index_querysets based on using value 0ede9ad
@acdha acdha PEP-8 4d199e0
@acdha acdha update_index: update all backends by default
* Change the default behaviour to update every backend
* Allow --using to be repeated to update multiple named backends
a018d7b
@acdha acdha clear_index: adopt new behaviour of update_index
* By default, all backends are cleared
* --using may be specified multiple times
1bfb090
@acdha acdha Basic tests for management commands with --using
This ensures that the management commands correctly process --using options and
pass them to the appropriate lower-level code
77146e6
@acdha

I was too quick to assume bugs: what looked like a regression at first glance was just my misreading a diff I hadn't looked at in awhile. Merge imminent…

@acdha acdha added a commit that closed this pull request Jan 24, 2013
@acdha acdha Pass active backend to index queryset calls (closes #534)
Now the Index index_queryset() and read_queryset() methods will be called with
the active backend name so they can optionally perform backend-specific
filtering.

This is extremely useful when using something like Solr cores to maintain
language specific backends, allowing an Index to select the appropriate
documents for each language::

    def index_queryset(self, using=None):
        return Post.objects.filter(language=using)

Changes:
    * clear_index, update_index and rebuild_index all default to processing
      *every* backend. ``--using`` may now be provided multiple times to select
      a subset of the configured backends.
    * Added examples to the Multiple Index documentation page
c5e0ce5
@acdha acdha closed this in c5e0ce5 Jan 24, 2013
@ntucker ntucker added a commit that referenced this pull request Jan 26, 2013
@ntucker ntucker Revert "Pass active backend to index queryset calls (closes #534)"
This reverts commit c5e0ce5.
e1a1d8c
@floppya floppya added a commit to floppya/django-haystack that referenced this pull request Mar 29, 2013
@acdha acdha Pass active backend to index queryset calls (closes #534)
Now the Index index_queryset() and read_queryset() methods will be called with
the active backend name so they can optionally perform backend-specific
filtering.

This is extremely useful when using something like Solr cores to maintain
language specific backends, allowing an Index to select the appropriate
documents for each language::

    def index_queryset(self, using=None):
        return Post.objects.filter(language=using)

Changes:
    * clear_index, update_index and rebuild_index all default to processing
      *every* backend. ``--using`` may now be provided multiple times to select
      a subset of the configured backends.
    * Added examples to the Multiple Index documentation page
779634c
@ntucker

Is it just me or does this break everything?

TypeError
index_queryset() got an unexpected keyword argument 'using'

@ntucker ntucker added a commit to ntucker/django-haystack that referenced this pull request Apr 4, 2013
@ntucker ntucker Revert "Pass active backend to index queryset calls (closes #534)"
This reverts commit c5e0ce5.
0228996
@acdha
@ntucker

Thanks acdha, somehow I missed that in upgrade documentation. :)

@acdha acdha deleted the acdha:multi-backend-using-extension branch May 3, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment