Skip to content
This repository has been archived by the owner on Aug 23, 2023. It is now read-only.

index refactor for more granular locks #1550

Merged
merged 11 commits into from
May 12, 2020

Conversation

replay
Copy link
Contributor

@replay replay commented Nov 25, 2019

Related to the issue #1539

This splits the main index lock up by introducing 3 new RW locks.

  1. Used to protect a new data structure called metaTagIdx which manages the meta tag data structures on a per org basis. This write lock only needs to be acquired when a new org starts using the meta tag index, so very rarely. metaTagIdx also implements all the exported index methods related to the meta tag index, it is embedded into the main index.
  2. The data structure metaTagIndex has been renamed to metaTagHierarchy to prevent confusion with the main meta tag index struct, and it has it's own RW lock now. It also has a few new accessor methods, because previously some of it's properties were directly accessed by other methods of the index package, but now all interactions go through the new accessors to make sure the lock is used correctly.
  3. The data structure metaTagRecords also has its own RW lock now, and a few other methods which previously accessed its properties directly are now using its accessor methods to ensure correct locking.

The method idsByTagQuery had to be refactored because when we add/delete meta records from the enricher we need to be able to execute a query on the tag index and pass the result into the enricher without releasing the read lock on the main index in-between, otherwise there would be a race condition. So the enricher now has a callback using which it can call idsByTagQuery on the main index. When executing such queries we now instantiate the result chan into which the query results will get pushed, pass it into idsByTagQuery which acquires the read lock on the main index and starts the query execution in a new routine, then we push the result chan into the enricher's event queue, and only release the read lock on the main index once the query has completed. The enricher's event handler then only needs to consume that result chan. That way we can still ensure consistency in the enricher.

@replay replay force-pushed the 20191125_index_refactor_for_more_granular_locks branch 6 times, most recently from 2fd863a to 50c3f5d Compare May 8, 2020 17:48
@replay replay force-pushed the 20191125_index_refactor_for_more_granular_locks branch from 50c3f5d to 8a3f98b Compare May 12, 2020 13:53
@replay replay merged commit b668b0d into master May 12, 2020
@replay replay deleted the 20191125_index_refactor_for_more_granular_locks branch May 12, 2020 14:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants