Count is not same in kibana and pandas

[abhimanyu3-zz]

I am getting data from elastic search to pandas using Search and scan method. My document count is in billions and millions. I noticed a weird thing. When I am matching the numbers in my pandas and kibana for the same time period the numbers are not the same. The greater the duration the more the difference i am getting. Sometimes its more in kibana and sometimes in pandas for the same time period. Is this normal? or it is happening because of the volume of data I am parsing?

[honzakral]

The problem with elasticsearch is that is updates the index only every so often (by default every second) when a refresh happens so the number might not be 100% accurate at any given moment.

Also since this is not an issue with elasticsearch-dsl python library I would like to ask you to open a topic for discussion on the more generic forum at https://discuss.elastic.co

Thank you!

[abhimanyu3-zz]

Thanks a lot for your input