Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix sort order on mixed date and date_nanos fields #43939

Closed
xeraa opened this issue Aug 25, 2019 · 3 comments · Fixed by #44212
Closed

Fix sort order on mixed date and date_nanos fields #43939

xeraa opened this issue Aug 25, 2019 · 3 comments · Fixed by #44212
Assignees
Labels
Feature:Discover Discover Application Team:Visualizations Visualization editors, elastic-charts and infrastructure

Comments

@xeraa
Copy link

xeraa commented Aug 25, 2019

Kibana version: 7.3.0

Elasticsearch version: 7.3.0

Server OS version: RedHat

Browser version: Brave 0.68.131, Chromium: 76.0.3809.100

Browser OS version: Mac OS X

Original install method (e.g. download page, yum, from source, etc.): Cloud

Describe the bug: When you are searching multiple indices and they are using a mix of date and date_nanos, the sort order is not correct. The reason is that the internal representation is a long since the start of the epoche for both, but because of the different precision this doesn't sort correctly. Elasticsearch has added a workaround for that in 7.2 with an option "numeric_type": "date_nanos" to cast everything to nanosecond precision.

Steps to reproduce:

  1. Mapping
PUT timestamp-millis
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date"
      }
    }
  }
}

PUT timestamp-nanos
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date_nanos"
      }
    }
  }
}
  1. Sample documents:
PUT timestamp-millis/_doc/1
{
  "timestamp": "2019-01-01T12:10:30.123Z"
}

PUT timestamp-millis/_doc/2
{
  "timestamp": "2019-01-01T12:10:30.124Z"
}

PUT timestamp-nanos/_doc/3
{
  "timestamp": "2019-01-01T12:10:30.123456789Z"
}

PUT timestamp-nanos/_doc/4
{
  "timestamp": "2019-01-01T12:10:30.123498765Z"
}
  1. Sort in Discover (format in the index pattern is switched to Date Nanos):

discover-nanos

Expected behavior: Sort correctly by using "numeric_type": "date_nanos" — probably only when the format of the index pattern is switched to Date Nanos.

Any additional context: I got curious after asked a question about it on Stack Overflow.

@timroes timroes added Feature:Discover Discover Application Team:Visualizations Visualization editors, elastic-charts and infrastructure labels Aug 26, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app

@kertal kertal self-assigned this Aug 26, 2019
@timroes
Copy link
Contributor

timroes commented Aug 26, 2019

We should only apply that flag to sort if we're sorting over date_nanos I think, since it can actually cause failures in regular date fields if you're using dates outside a specific range, as states by the docs.

@kertal
Copy link
Member

kertal commented Aug 26, 2019

@xeraa Thx for the numeric_type hint!
@timroes agreed!
We also have to investigate Index Pattern creation for the mixed case aka date format melange

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Discover Discover Application Team:Visualizations Visualization editors, elastic-charts and infrastructure
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants