Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Discover] Field statistics request fails for epoch_second timestamp mappings #148140

Closed
kertal opened this issue Dec 28, 2022 · 5 comments · Fixed by #148288
Closed

[Discover] Field statistics request fails for epoch_second timestamp mappings #148140

kertal opened this issue Dec 28, 2022 · 5 comments · Fixed by #148288
Assignees
@jughosta
Labels
bug Fixes for quality problems that affect the customer experience Feature:Discover Discover Application Feature:UnifiedFieldList The unified field list component used by Lens & Discover impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
Discover

Comments

@kertal
Copy link
Member

kertal commented Dec 28, 2022

Kibana version:
8.5.x

Describe the bug:
When there's a mapping like epoch_second used for the timefield used in the data view, the requests returning the field statistics of Discover's field list fail:

failed to parse date field [2007-12-28T08:28:29.697Z] with format [epoch_second]: [failed to parse date field [2007-12-28T08:28:29.697Z] with format [epoch_second]]"

The requests for the statistics needs to have "format": "strict_date_optional_time" in the range to work correctly

Steps to reproduce:

  1. Create demo data with a timestamp like this "timestamp": { "type": "date", "format": "epoch_second" }
  2. Go to Discover and create an data view for the demo data
  3. When showing the field popover there's just a message No field data for the current search.
    Discover_-_Elastic

Expected behavior:
In the popover there should be a list of the Top10 field values
@kertal kertal added bug Fixes for quality problems that affect the customer experience Feature:Discover Discover Application loe:small Small Level of Effort impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) Feature:UnifiedFieldList The unified field list component used by Lens & Discover labels Dec 28, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kibanamachine kibanamachine added this to Inbox in Discover Dec 28, 2022
@jughosta jughosta self-assigned this Jan 3, 2023
@jughosta jughosta mentioned this issue Jan 3, 2023
Merged
@jughosta
Copy link
Contributor

jughosta commented Jan 3, 2023

Thanks, @kertal, for catching it!

Discover automation moved this from Inbox to Done Jan 5, 2023
jughosta added a commit that referenced this issue Jan 5, 2023
@jughosta
[Discover] Fix field stats for epoch time format (#148288)
1b3769c 
Closes #148140

## Summary

This PR fixes time format so it can work also for epoch.

Before:
<img width="400" alt="Screenshot 2023-01-03 at 12 33 17"
src="https://user-images.githubusercontent.com/1415710/210350243-b3ae1d1a-4c76-46e7-930f-0ca9835dc121.png">

After:
<img width="400" alt="Screenshot 2023-01-03 at 12 32 49"
src="https://user-images.githubusercontent.com/1415710/210350274-af80f558-6a32-4da6-9cb9-857a8cb3de70.png">

Sample data for testing:
```
PUT test
PUT test/_mapping
{
    "properties": {
        "timestamp": {
            "type": "date",
            "format": "epoch_second"
        },
        "message":  {
          "type": "keyword"
        },
        "bytes": {
          "type": "long"
        }
    }
}
    
POST test/_doc/1
{
    "timestamp": 1669912088.9230318,
    "message": "Message 1",
    "bytes": 250
}
POST test/_doc/2
{
    "timestamp": 1669912088.9230319,
    "message": "Message 2",
    "bytes": 20
}
```
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Jan 5, 2023
@jughosta
[Discover] Fix field stats for epoch time format (elastic#148288)
d780ffc 
Closes elastic#148140

## Summary

This PR fixes time format so it can work also for epoch.

Before:
<img width="400" alt="Screenshot 2023-01-03 at 12 33 17"
src="https://user-images.githubusercontent.com/1415710/210350243-b3ae1d1a-4c76-46e7-930f-0ca9835dc121.png">

After:
<img width="400" alt="Screenshot 2023-01-03 at 12 32 49"
src="https://user-images.githubusercontent.com/1415710/210350274-af80f558-6a32-4da6-9cb9-857a8cb3de70.png">

Sample data for testing:
```
PUT test
PUT test/_mapping
{
    "properties": {
        "timestamp": {
            "type": "date",
            "format": "epoch_second"
        },
        "message":  {
          "type": "keyword"
        },
        "bytes": {
          "type": "long"
        }
    }
}

POST test/_doc/1
{
    "timestamp": 1669912088.9230318,
    "message": "Message 1",
    "bytes": 250
}
POST test/_doc/2
{
    "timestamp": 1669912088.9230319,
    "message": "Message 2",
    "bytes": 20
}
```

(cherry picked from commit 1b3769c)
kibanamachine added a commit that referenced this issue Jan 5, 2023
@kibanamachine @jughosta
[8.6] [Discover] Fix field stats for epoch time format (#148288) (#14…
57f91bf 
…8457)

# Backport

This will backport the following commits from `main` to `8.6`:
- [[Discover] Fix field stats for epoch time format
(#148288)](#148288)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Rechkunova","email":"julia.rechkunova@elastic.co"},"sourceCommit":{"committedDate":"2023-01-05T14:52:29Z","message":"[Discover]
Fix field stats for epoch time format (#148288)\n\nCloses
#148140
Summary\r\n\r\nThis PR fixes time format so it can work also for
epoch.\r\n\r\nBefore:\r\n<img width=\"400\" alt=\"Screenshot 2023-01-03
at 12 33
17\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350243-b3ae1d1a-4c76-46e7-930f-0ca9835dc121.png\">\r\n\r\nAfter:\r\n<img
width=\"400\" alt=\"Screenshot 2023-01-03 at 12 32
49\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350274-af80f558-6a32-4da6-9cb9-857a8cb3de70.png\">\r\n\r\nSample
data for testing:\r\n```\r\nPUT test\r\nPUT test/_mapping\r\n{\r\n
\"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"message\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"bytes\": {\r\n \"type\": \"long\"\r\n }\r\n
}\r\n}\r\n \r\nPOST test/_doc/1\r\n{\r\n \"timestamp\":
1669912088.9230318,\r\n \"message\": \"Message 1\",\r\n \"bytes\":
250\r\n}\r\nPOST test/_doc/2\r\n{\r\n \"timestamp\":
1669912088.9230319,\r\n \"message\": \"Message 2\",\r\n \"bytes\":
20\r\n}\r\n```","sha":"1b3769c86b683a3e1762f23b6f503479957044c6","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:DataDiscovery","backport:prev-minor","Feature:UnifiedFieldList","v8.7.0"],"number":148288,"url":"#148288
Fix field stats for epoch time format (#148288)\n\nCloses
#148140
Summary\r\n\r\nThis PR fixes time format so it can work also for
epoch.\r\n\r\nBefore:\r\n<img width=\"400\" alt=\"Screenshot 2023-01-03
at 12 33
17\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350243-b3ae1d1a-4c76-46e7-930f-0ca9835dc121.png\">\r\n\r\nAfter:\r\n<img
width=\"400\" alt=\"Screenshot 2023-01-03 at 12 32
49\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350274-af80f558-6a32-4da6-9cb9-857a8cb3de70.png\">\r\n\r\nSample
data for testing:\r\n```\r\nPUT test\r\nPUT test/_mapping\r\n{\r\n
\"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"message\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"bytes\": {\r\n \"type\": \"long\"\r\n }\r\n
}\r\n}\r\n \r\nPOST test/_doc/1\r\n{\r\n \"timestamp\":
1669912088.9230318,\r\n \"message\": \"Message 1\",\r\n \"bytes\":
250\r\n}\r\nPOST test/_doc/2\r\n{\r\n \"timestamp\":
1669912088.9230319,\r\n \"message\": \"Message 2\",\r\n \"bytes\":
20\r\n}\r\n```","sha":"1b3769c86b683a3e1762f23b6f503479957044c6"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"#148288
Fix field stats for epoch time format (#148288)\n\nCloses
#148140
Summary\r\n\r\nThis PR fixes time format so it can work also for
epoch.\r\n\r\nBefore:\r\n<img width=\"400\" alt=\"Screenshot 2023-01-03
at 12 33
17\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350243-b3ae1d1a-4c76-46e7-930f-0ca9835dc121.png\">\r\n\r\nAfter:\r\n<img
width=\"400\" alt=\"Screenshot 2023-01-03 at 12 32
49\"\r\nsrc=\"https://user-images.githubusercontent.com/1415710/210350274-af80f558-6a32-4da6-9cb9-857a8cb3de70.png\">\r\n\r\nSample
data for testing:\r\n```\r\nPUT test\r\nPUT test/_mapping\r\n{\r\n
\"properties\": {\r\n \"timestamp\": {\r\n \"type\": \"date\",\r\n
\"format\": \"epoch_second\"\r\n },\r\n \"message\": {\r\n \"type\":
\"keyword\"\r\n },\r\n \"bytes\": {\r\n \"type\": \"long\"\r\n }\r\n
}\r\n}\r\n \r\nPOST test/_doc/1\r\n{\r\n \"timestamp\":
1669912088.9230318,\r\n \"message\": \"Message 1\",\r\n \"bytes\":
250\r\n}\r\nPOST test/_doc/2\r\n{\r\n \"timestamp\":
1669912088.9230319,\r\n \"message\": \"Message 2\",\r\n \"bytes\":
20\r\n}\r\n```","sha":"1b3769c86b683a3e1762f23b6f503479957044c6"}}]}]
BACKPORT-->

Co-authored-by: Julia Rechkunova <julia.rechkunova@elastic.co>
@jughosta jughosta mentioned this issue Jan 9, 2023
Closed
@derek-diaz
Copy link

derek-diaz commented Jan 12, 2023 •
edited

@matschaffer @jughosta This issue is still happening.

I upgraded my Elastic Cloud production cluster to v8.6.0 and the issue still persists if discover:showLegacyFieldTopValues disabled:

image

image

When I enable the feature flag discover:showLegacyFieldTopValues then the field data populates correctly:

image

image

I have an open ticket with Elastic Support for this same issue that is still open after 2 months (01146753).

@davismcphee
Copy link
Contributor

@derek-diaz According to this comment, it looks like the fix was not merged in time for the v8.6.0 release and will instead be included in the v8.6.1 release. In the meantime, discover:showLegacyFieldTopValues: true is the necessary workaround.

@derek-diaz
Copy link

@derek-diaz According to this comment, it looks like the fix was not merged in time for the v8.6.0 release and will instead be included in the v8.6.1 release. In the meantime, discover:showLegacyFieldTopValues: true is the necessary workaround.

Ah that makes sense, the elastic support team told me that v8.6.0 had the fix. Thanks for the clarification! 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jughosta jughosta

Labels
bug Fixes for quality problems that affect the customer experience Feature:Discover Discover Application Feature:UnifiedFieldList The unified field list component used by Lens & Discover impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
No open projects
Discover
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Discover] Fix field stats for epoch time format jughosta/kibana
5 participants
@kertal @jughosta @derek-diaz @elasticmachine @davismcphee