Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'gt' gives unexpected results. #43277

Closed
Sanjaygk opened this issue Jun 17, 2019 · 3 comments

Comments

@Sanjaygk
Copy link

commented Jun 17, 2019

Elasticsearch version: 7.0.0

OS version (uname -a if on a Unix-like system):
Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
The elasticsearch query,

GET my_index*/_search
{
  "query":                                 
    {"range": {"timestamp": {"gt": "now-1M"}}}
}

returns,

{
  "took" : 923,
  "timed_out" : false,
  "_shards" : {
    "total" : 14,
    "successful" : 14,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Returns zero hits, but if i do change the range to "now-2w" it returns hits. this is quite strange! and the relation too is specified as 'eq' rather than 'gt' in above response.

it works well with 'gte' though.

Steps to reproduce:

  1. Try to query data for different time periods (week, month) using console with 'gt' parameter on elasticsearch 7.0.0. For some time periods, it'll return zero hits and 'relation' as 'eq' even though there's data for that time period in elasticsearch.

I don't see any errors in elasticsearch logs. (Discover tab in Kibana shows proper results as expected)

@elasticmachine

This comment has been minimized.

Copy link
Collaborator

commented Jun 17, 2019

@DaveCTurner

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2019

Relates https://discuss.elastic.co/t/search-api-returns-zero-hits/185963 /cc @henningandersen.

This behaviour looks to be deliberate, although I will admit that I find it surprising too. The gt vs gte applies to the integer number of time periods added or subtracted, so "gt": "now-2w" means the same as "gte": "now-1w" and, crucially, only returns hits from the last 7 days. Similarly, "gt": "now-1M" means the same as "gte": "now-0M" and thus does not return any hits from the past at all.

Labelling this for team discussion in case my confusion here is misplaced.

jimczi added a commit to jimczi/elasticsearch that referenced this issue Jun 17, 2019

Fix round up of date range without rounding
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes elastic#43277
@jimczi

This comment has been minimized.

Copy link
Member

commented Jun 17, 2019

Thanks for reporting @Sanjaygk. This is an actual bug that was introduced with the move to Java Time. The logic that we apply to exclude rounded values (now/M) is also applied to simple operation like now-1M in the Java date math parser. So instead of adding 1ms to get the values greater than now-1M we also add 1M to the result of the operation. I opened #43303 to restore the original behavior.

@jimczi jimczi added >bug and removed team-discuss labels Jun 17, 2019

jimczi added a commit that referenced this issue Jun 20, 2019

Fix round up of date range without rounding (#43303)
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes #43277

jimczi added a commit that referenced this issue Jun 20, 2019

Fix round up of date range without rounding (#43303)
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes #43277

jimczi added a commit that referenced this issue Jun 20, 2019

Fix round up of date range without rounding (#43303)
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes #43277

jkakavas added a commit to jkakavas/elasticsearch that referenced this issue Jun 27, 2019

Fix round up of date range without rounding (elastic#43303)
Today when searching for an exclusive range the java date math parser rounds up the value
with the granularity of the operation. So when searching for values that are greater than
"now-2M" the parser rounds up the operation to "now-1M". This behavior was introduced when
we migrated to java date but it looks like a bug since the joda math parser rounds up values
but only when a rounding is used. So "now/M" is rounded to "now-1ms" (minus 1ms to get the largest inclusive value)
in the joda parser if the result should be exclusive but no rounding is applied if the input
is a simple operation like "now-1M". This change restores the joda behavior in order to have
a consistent parsing in all versions.

Closes elastic#43277
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.