Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

auto_update class attribute added for AbstractWhoosheer #19

Merged
merged 1 commit into from
Aug 1, 2017

Conversation

clime
Copy link
Contributor

@clime clime commented Jul 20, 2016

Decides if a whoosheer should be updated by on_commit method.

Default value is True and it can be changed in a user-specified whoosheer:

@whooshee.register_whoosheer
class CoprWhoosheer(AbstractWhoosheer):
   schema = whoosh.fields.Schema(
        copr_id=whoosh.fields.NUMERIC(stored=True, unique=True)
   )

  auto_update = False

The rationale for this feature is that index updates in our system can take a long time (around 0.8s) and if there are several concurrent requests to update the search index, some of them occasionally time out with 'LockError'. That's why we need to switch from continuous auto-updating of the search index to updating in batches (every hour, e.g.) and only for those records that have been modified (this is covered by a piece of custom logic).

The optimization here is that if a db record was modified many times in the passed hour, the search index record will be modified only once. Also search index updating will become sequential and hence proof to lock errors caused by many concurrent accesses.

Our index has become very large lately and that is why we need this. I'll add README and tests if you will agree with this feature.

@bkabrda
Copy link
Collaborator

bkabrda commented Aug 12, 2016

Sorry for responding so late to this issue, I didn't notice the notification in my inbox...

Yeah, this makes perfect sense to me. If you can provide docs and tests, it will be absolutely awesome (feel free to describe the whole usecase in README, so that readers understand why this was implemented and what problem it solves).

Decides if a whoosheer should be updated by whooshee.on_commit method.
@clime
Copy link
Contributor Author

clime commented Jan 23, 2017

Docs & tests added.

@bkabrda
Copy link
Collaborator

bkabrda commented Aug 1, 2017

LGTM, finally merging! :) thanks and sorry for the delay.

@bkabrda bkabrda merged commit 6515b17 into fedora-copr:master Aug 1, 2017
@bkabrda
Copy link
Collaborator

bkabrda commented Aug 7, 2017

Released as part of flask-whooshee 0.5.0: https://pypi.python.org/pypi/flask-whooshee/0.5.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants