Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

smarter ordering for queries that don't use zk first #935

Merged
merged 3 commits into from Mar 10, 2016

Conversation

@ssalinas
Copy link
Member

ssalinas commented Mar 4, 2016

Currently, when searching anything where queryUsesZkFirst returns false, all results from zk are still initially in our list of returned items before fetching from history. So, after sorting, regardless of page, the zk results will likely be on the page and much of the persisted history will not be shown. In short, anything that hasn't been persisted yet will likely show up on every page of task search results.

This PR implements two things:

  • taskHistoryQueryUsesZkFirst option -> if true this would have every query for task history use zk items first. This will only require a single history query, and thus be much faster, with the caveat that anything not persisted will always appear before anything that has already persisted in the general order (but paging will be correct)
  • If taskHistoryQueryUsesZkFirst is false we will make similar initial queries for items from zk and from persisted history. But, after ordering them, we will check that we have fetched all relevant items from the history. If we have not (i.e. some non-persisted items in the limitStart/limitCount range are beyond the range of all persisted items we have fetched), then we will fetch additional items to ensure correct ordering of all items, but at the added cost of additional queries needed.

My overall thought on this was that we can at least give the user the choice of efficiency/speed vs ensuring correct ordering.

Only other thing I wasn't sure on was whether to make taskHistoryQueryUsesZkFirst available on the query as well to give each individual user making a query the option as well as being abel to set the system-wide default.

/cc @wsorenson @tpetr @Calvinp

ssalinas added 2 commits Mar 4, 2016
@ssalinas ssalinas added the hs_staging label Mar 7, 2016
@ssalinas ssalinas added the hs_qa label Mar 8, 2016
@ssalinas
Copy link
Member Author

ssalinas commented Mar 10, 2016

Going to merge this into task_search since both are now in qa branch

ssalinas added a commit that referenced this pull request Mar 10, 2016
smarter ordering for queries that don't use zk first
@ssalinas ssalinas merged commit fefb101 into task_search Mar 10, 2016
0 of 2 checks passed
0 of 2 checks passed
continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/push The Travis CI build failed
Details
@ssalinas ssalinas deleted the task_search_zk_first branch Mar 10, 2016
@ssalinas ssalinas modified the milestone: 0.4.12 Apr 1, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant
You can’t perform that action at this time.