New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stop indexing activity on ddoc update #635

Merged
merged 1 commit into from Jul 31, 2017

Conversation

Projects
None yet
2 participants
@mayya-sharipova
Contributor

mayya-sharipova commented Jul 4, 2017

Currently when ddoc is updated, couch_index and couch_index_updater processes
corresponding to the previous version of ddoc will still exist until
all indexing processing initiated by them is done.
When ddoc of a big database is rapidly modified, this puts a lot
of unnecessary strain on database.

With this change, when ddoc is updated:

  • all couch_index processes for the previous version of ddoc will be shutdown
  • all linked to them couch_index_updater processes will die as well
  • all processes waiting for indexing activity to be finished (waiters
    for couch_index:get_status) will receive an immediate reply:
    ddoc_updated. Interactive user requests (view queries) will get response:
    {404, <<"not_found">>, <<"Design document was updated or deleted.">>}

Overview

When design document is updated, stop couch_index processes for the previous version
of the design doc.

Testing recommendations

make check apps=couch_mrview tests=ddoc_update_test_

GitHub issue number

Related Pull Requests

Checklist

  • Code is written and works correctly;
  • Changes are covered by tests;
  • Documentation reflects the changes;
@davisp

I'd like to try figuring out a fix to shutting down indexes that have multiple design documents associated since that is an advocated pattern for production apps.

The only other thing is that the new RPC messages were at first worrisome but thinking on it we'd only hit that for a design document updated during an upgrade so I'm fine leaving that window open (ie, crashing any old coordinators if someone does modify a design document during upgrade).

@davisp

Awesome! This is so much better than the approach I was mulling over trying to store design doc ids in the individual indexer states. Just one minor tweak to the add_to_ets message handling and this will be good to go.

@davisp

davisp approved these changes Jul 25, 2017

+1

Awesome work! Looks like you need to fix a merge conflict and then you can squerge. Though I'd hold off merging till after @wohali branches the 2.1.x branch so we're not adding churn just before the release.

Mayya Sharipova
Stop couch_index processes on ddoc update
Currently when ddoc is updated, couch_index and couch_index_updater processes
corresponding to the previous version of ddoc will still exist until
all indexing processing initiated by them is done.
When ddoc of a big database is rapidly  modified, this puts a lot
of unnecessary strain on database resources.

With this change, when ddoc is updated:
* all couch_index processes for the previous version of ddoc will be shutdown
* all linked to them couch_index_updater processes will die as well
* all processes waiting for indexing activity to be finished (waiters
    for couch_index:get_status) will receive an immediate reply:
    ddoc_updated. Interactive user requests (view queries) will get response:
    {404, <<"not_found">>, <<"Design document was updated or deleted.">>}

Check if there are ddocs that use the same couch_index process
before closing it on ddoc_updated

1. When opening an index, always add a record {DbName, {DDocId, Sig}} to ?BY_DB.
2. When ddoc_updated, check if there other ddocs in ?BY_DB with the same Sig.
    If there are no, stop couch_index processes.
    If there are other, only remove {DbName, {DDocId, Sig}}
    record from ?BY_DB for this ddoc.

@davisp davisp merged commit 2e7ca45 into apache:master Jul 31, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment