smarter ordering for queries that don't use zk first #935
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when searching anything where
queryUsesZkFirst
returnsfalse
, all results from zk are still initially in our list of returned items before fetching from history. So, after sorting, regardless of page, the zk results will likely be on the page and much of the persisted history will not be shown. In short, anything that hasn't been persisted yet will likely show up on every page of task search results.This PR implements two things:
taskHistoryQueryUsesZkFirst
option -> iftrue
this would have every query for task history use zk items first. This will only require a single history query, and thus be much faster, with the caveat that anything not persisted will always appear before anything that has already persisted in the general order (but paging will be correct)taskHistoryQueryUsesZkFirst
isfalse
we will make similar initial queries for items from zk and from persisted history. But, after ordering them, we will check that we have fetched all relevant items from the history. If we have not (i.e. some non-persisted items in the limitStart/limitCount range are beyond the range of all persisted items we have fetched), then we will fetch additional items to ensure correct ordering of all items, but at the added cost of additional queries needed.My overall thought on this was that we can at least give the user the choice of efficiency/speed vs ensuring correct ordering.
Only other thing I wasn't sure on was whether to make
taskHistoryQueryUsesZkFirst
available on the query as well to give each individual user making a query the option as well as being abel to set the system-wide default./cc @wsorenson @tpetr @Calvinp