Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Optimise nextFetchDate to speed up queries to Elasticsearch #452
The queries in the spouts take longer as the status index gets larger. We use sampling with the aggregation spouts but could also optimise the speed by querying on a nextFetchDate value which would be old enough to minimise the amount of docs on which the aggregations run, while retrieving as many docs as possible.
The same approach could also be used for the collapsing spout.