Only return aggregations on the first page with scroll and forbidden with scan

[jpountz]

Aggregations are collection-wide statistics over one or several indices so:

• scan should not be allowed to use aggregations since it never collects all documents at once,
• scroll should only return aggregations on the first page (see Facets incorrect when scrolling #1642)

[jpountz]

For reference, I plan to add documentation for it, but I'm first looking for feedback to know whether this is the correct approach.

[martijnvg]

@jpountz This approach looks good to me. I think this issue should be marked as breaking as well, since for normal scroll we did include the aggs also in subsequent responses.

[jpountz]

There is one open question about the behavior with search_type=scan: scan works by collecting just enough documents on each round while aggs work by collecting all documents in one go. So I don't think there is some way around collecting matches twice (once for aggs and once for the scan). So potentially there are two options:

1. prevent the usage of aggs with scan (what the current PR does)
2. or collect matches twice: once in the first call purely for aggregations, and then in a streaming fashion for the scan.

I like the fact that the first option doesn't do any magic and makes sure that scan is cheap in all cases but on the other hand, 2. could make scan/scroll more consistent with normal scroll (aggs returned in the first page and ignored in subsequent pages).

What do you think?

[jpountz]

Side note: assuming 1. is implemented, 2. could be done from client side with negligible overhead (one round trip) by running first a count request to compute aggs and then a scan request to get hits back.

[costin]

As a directly interested client :), I'll like to understand how the two requests would be an alternative to 2 (ideally considering the preference api as well).

[s1monw]

I personally like the fact that scan is simply a stream and I think we should keep it that way. IMO you can do the entire request in 2 steps (one for aggs and one for scan) this essentially means that you can potentially get slightly outdated aggregations but I think that is an acceptable solution. Scan requests should not support aggregations at all.

[jpountz]

@costin I think you can have two requests by having one that is a SCAN and only fetches hits while another query would be of type COUNT and would compute aggregations. I don't think the preference API raises particular issues: if you use it to go to particular shards, that actually acts as a filter on the documents that match, so I think that is fine?

[clintongormley]

I'm happy with scan not supporting aggs, and scroll-without-scan just returning aggs on the first request.