Count total

[erictuvesson]

I would like to do a count call and get the total amount of matching documents, problem is that search_flip is adding size and offset to all requests. Is there a way I can remove that?

MyIndex.where(field_id: 12).request
=> {:query=>{:bool=>{:filter=>[{:term=>{:field_id=>12}}]}}, :from=>0, :size=>30}

MyIndex.where(field_id: 12).count
=> 30

When I do a call like

curl http://0.0.0.0:9200/my_index/_count -d '{"query":{"bool":{"filter":[{"term":{"field_id":12}}]}}}' -H 'Content-Type: application/json'

I get the total amount in return

{"count":36757285,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}

[mrkamel]

Use .total_count or .total_entries like stated in the readme and reference docs
https://www.rubydoc.info/github/mrkamel/search_flip/SearchFlip/Response#total_entries-instance_method

MyIndex.where(field_id: 12).total_count

[erictuvesson]

Awesome, thanks!

I tried to find anything about it in the readme, just I was looking more for count and total_count is not there.

[erictuvesson]

It still doesn't work in the way I expect.

irb(main):006:0> Index.range(:created_at_date, gt: date, lte: date + 1.day).total_entries
=> 10000
irb(main):007:0> Index.where(master_module_id: 12).total_entries
=> 10000
irb(main):008:0> Index.where(master_module_id: 12).total_count  
=> 10000

I looked into the code and it's still doing the same search query where I want to do a count query.

https://github.com/mrkamel/search_flip/blob/master/lib/search_flip/response.rb#L42

Elasticsearch (Count API)
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-count.html

[mrkamel]

search flip is not using the COUNT API, because then you would always need two queries: one to get the results, and one to get the count.

ES always returns 10_000, because since version 7, ES by default stops counting from 10_000 on for performance reasons. To get the exact number of results for ES > 7, you have to add track_total_hits(true), e.g.

Index.where(...).track_total_hits(true).total_count

[erictuvesson]

Okay, I was aware of the 10_000 limit, just not aware of the API call to make it all visible.

irb(main):005:0> Index.range(:created_at_date, gt: date, lte: date + 1.day).where(master_module_id: 12).track_total_hits(true).total_count
=> 1490200

Thanks for the help again!