Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache date range filters that use now with rounding #4947

Closed
clintongormley opened this issue Jan 30, 2014 · 6 comments
Closed

Cache date range filters that use now with rounding #4947

clintongormley opened this issue Jan 30, 2014 · 6 comments
Labels
>bug v0.90.11 v1.0.0.RC2 v1.1.0 v2.0.0-beta1

Comments

@clintongormley
Copy link

In #4846, caching of date range filters which use now was disabled. However, that only makes sense as long as there is no rounding happening.

For instance, this filter does make sense to cache:

{
  "range": {
      "date": {
          "gt":  "now/d"
       }
   }
}
@uboness
Copy link
Contributor

uboness commented Jan 30, 2014

good one :)

@bleskes
Copy link
Contributor

bleskes commented Jan 30, 2014

It depends. Wouldn't it be just as confusing around midnight? Where just after midnight you might not see data? I agree it's much less of an issue but in certain scenarios you may still get wrong results.

On Thu, Jan 30, 2014 at 4:46 AM, uboness notifications@github.com wrote:

good one :)

Reply to this email directly or view it on GitHub:
#4947 (comment)

@clintongormley
Copy link
Author

@bleskes This isn't about wrong results - it's about cache churn. The filter is cached with whatever now evaluates to. Typically filtering on eg now - 1h makes sense not to cache, because you'll never reuse that value.

But a filter with now/d likely will be reused. Think about this pattern when retrieving logs for the last hour in an index containing logs for the whole month:

"bool": {
    "must": [
        { "range" : {
            "timestamp" : {
                "gt" : "now/d" 
            }
        },
        { "range" : {
            "timestamp" : {
                "gt" : "now-1h" 
            }
        }
    ]
}

You want the first filter to be cached, and the second filter not to be cached.

@imotov
Copy link
Contributor

imotov commented Jan 30, 2014

Could we resolve the expression with now in it using the current time and make it part of the cache key?

@martijnvg
Copy link
Member

That is what used to happen, but since now is different every time, the next time now is used the previous cache key wouldn't be used, this results in thrashing the filter cache (adding entries that never end up being used).

@bleskes
Copy link
Contributor

bleskes commented Jan 30, 2014

@clintongormley misunderstood then (and learned something about how our internal caching works: @imotov - the caching logic compares the term the expression resolves to)

@martijnvg martijnvg mentioned this issue Jan 30, 2014
Closed
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Jan 30, 2014
@martijnvg
The forceful no cache behaviour for range filter with now date match …
c59ad51 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes elastic#4947
 Relates to elastic#4846
martijnvg added a commit that referenced this issue Jan 30, 2014
@martijnvg
The forceful no cache behaviour for range filter with now date match …
b8ea97b 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes #4947
 Relates to #4846
martijnvg added a commit that referenced this issue Jan 30, 2014
@martijnvg
The forceful no cache behaviour for range filter with now date match …
436d4b5 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes #4947
 Relates to #4846
martijnvg added a commit that referenced this issue Jan 30, 2014
@martijnvg
The forceful no cache behaviour for range filter with now date match …
dc5820c 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes #4947
 Relates to #4846
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@martijnvg
The forceful no cache behaviour for range filter with now date match …
d6a9c1c 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes elastic#4947
 Relates to elastic#4846
mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015
@martijnvg
The forceful no cache behaviour for range filter with now date match …
37a56ed 
…expression should only be active if no rounding has been specified for `now` in the date range range expression (for example: `now/d`).

Also the automatic now detection in range filters is overrideable by the `_cache` option.

 Closes elastic#4947
 Relates to elastic#4846
@crespocarlos crespocarlos mentioned this issue Apr 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
>bug v0.90.11 v1.0.0.RC2 v1.1.0 v2.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Range filter no cache behaviour for now with rounding martijnvg/elasticsearch
5 participants
@clintongormley @uboness @martijnvg @imotov @bleskes