Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Index patterns] Store field-specific metadata from mappings #82318

Closed
wylieconlon opened this issue Nov 2, 2020 · 16 comments · Fixed by #174973
Closed

[Index patterns] Store field-specific metadata from mappings #82318

wylieconlon opened this issue Nov 2, 2020 · 16 comments · Fixed by #174973
Labels
enhancement New value added to drive a business result Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort papercut Small "burr" in the product that we should fix. Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
kibana-app-arch Discover

Comments

@wylieconlon
Copy link
Contributor

Elasticsearch now has standard metadata entries which we expect to use throughout the stack. The first step to integrating this in Kibana is to store field meta information which is part of field caps in the Kibana Index Pattern object.

cc @mattkime
@wylieconlon wylieconlon added enhancement New value added to drive a business result Feature:Data Views Data Views code and UI - index patterns before 8.0 Team:AppArch labels Nov 2, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-arch (Team:AppArch)

@kibanamachine kibanamachine added this to To triage in kibana-app-arch Nov 2, 2020
@mattkime
Copy link
Contributor

mattkime commented Nov 2, 2020

@wylieconlon Thanks for bringing this to my attention. Its definitely something we should do. Can you help me figure out the priority? What would be built on top of this and is it roadmapped?

@wylieconlon wylieconlon mentioned this issue Nov 2, 2020
Open
@wylieconlon
Copy link
Contributor Author

@mattkime I added some links to this issue from the related issues. Now that the decisions is made at the stack level, solutions like Observability are planning on adding metadata in a few places as early as 7.11. The most basic implementation I would hope for is that we would assign a different field formatter when the metadata indicates the type

This was referenced Nov 2, 2020
Closed
Closed
@flash1293 flash1293 mentioned this issue Mar 26, 2021
Open
@timroes timroes mentioned this issue Apr 30, 2021
Closed
@exalate-issue-sync exalate-issue-sync bot added impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Jun 21, 2021
@petrklapka petrklapka added Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) and removed Team:AppServicesSv labels Nov 28, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kibanamachine kibanamachine added this to Inbox in Discover Nov 28, 2022
@kertal kertal added the papercut Small "burr" in the product that we should fix. label Feb 16, 2023
@kertal
Copy link
Member

kertal commented Feb 16, 2023

@mattkime would be interesting to think about the size/effort, sounds more like hours?

@dej611
Copy link
Contributor

dej611 commented Jan 16, 2024

Just talked with @felixbarny and it would be good to have the meta.unit field leveraged for field formatting in Kibana.

@kertal
Copy link
Member

kertal commented Jan 16, 2024

so the request would be to apply auto field formatting given the meta.unit fields was available via field_caps request is this correct? FYI @mattkime @davismcphee

@dej611
Copy link
Contributor

dej611 commented Jan 16, 2024

Yes, it is correct.

@davismcphee
Copy link
Contributor

It sounds like a reasonable ask to me 👍 It would be convenient for users to only have to configure this once in ES instead of having to reconfigure it on the Kibana side too.

@mattkime
Copy link
Contributor

@dej611 @felixbarny Is there an example data set you're using? Whats the best way for me to find a meta.unit value?

@felixbarny
Copy link
Member

This page documents the conventions around meta.unit: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-field-meta.html. I'd suggest to just create a very simple mapping/test dataset using these conventions.

In this Elasticsearch PR, I'm proposing to prefer UCUM symbols but it keeps compatibility with the existing conventions. So after that PR has been merged, we should also update Kibana accordingly.

@mattkime
Copy link
Contributor

mattkime commented Jan 16, 2024 •
edited
Loading

field caps returns an array for meta.unit - is it okay to just take the first value?

How should we handle a different formats are supplied by different indices? aka format conflicts.

This PR makes field meta content available - #174973 - more work needs to be done to make it useful for formatters.

@felixbarny Can you supply a pair list of unit strings and the cooresponding kibana formatters that need to be applied?

@felixbarny
Copy link
Member

I'm not really familiar with Kibana formatters. Could you please propose a table of the mappings from meta.unit conventions as documented here to Kibana formatters? Thanks!

@felixbarny
Copy link
Member

How should we handle a different formats are supplied by different indices? aka format conflicts.

Good question. I think if there are conflicts, we shouldn't use a formatter by default.

@felixbarny
Copy link
Member

To expand on this a little more:

We should have a special case for the documented conventions for meta.unit: percent (1=100%), byte or time unit.
But in essence, the unit is free form, so it could be any string.

For the well-known units, we should do a bit more than just appending the unit. We should convert them to an appropriate unit of the same scale, depending on the magnitude of the value. For example, if meta.unit=nanos and the value is 10000000000, we should display that as 10s. Similar for byte. If it’s a big value, we should display it as KB/MB/GB/TB (maybe KiB etc.? I have no strong opinion on that).

For units that aren’t well-known, we can just append the unit.

I think we should also special-case two other UCUM (non-)units:

  • Non-units: UCUM specifies non-units to be in curly braces expressions. For example, {requests} or {threads}. When appending that unit to the values, we should display that without the curly braces, such as as requests or threads.
  • The default unit 1 (the unity). We should treat this as if the unit had not been set.

We can leave out the special casing of UCUM and do that in a follow-up PR, once elastic/elasticsearch#104037 is merged in which we recommend using UCUM symbols, which we're doing at the moment.

@felixbarny
Copy link
Member

I talked to @mattkime and we agreed that we should leave out the non-unit things for now.

@mattkime is there anything else that needs to be clarified or is this ready for implementation now?

@mattkime mattkime mentioned this issue Jan 30, 2024
Merged
mattkime added a commit that referenced this issue Feb 6, 2024
@mattkime
[data views] Default field formatters based on field meta values (#17…
cde8f2b 
…4973)

## Summary

Default field formatters based on field meta units data.

Note: the smallest unit our formatter will show is milliseconds which
means micro and nanoseconds may round down to zero for smaller values.
#176112


Closes: #82318

Mapping and doc setup for testing - 

```
PUT my-index-000001

PUT my-index-000001/_mapping
{
    "properties": {
            "nanos": {
        "type": "long",
        "meta": {
          "unit": "nanos"
        }
      },
            "micros": {
        "type": "long",
        "meta": {
          "unit": "micros"
        }
      },
      "ms": {
        "type": "long",
        "meta": {
          "unit": "ms"
        }
      },
                  "second": {
        "type": "long",
        "meta": {
          "unit": "s"
        }
      },
                  "minute": {
        "type": "long",
        "meta": {
          "unit": "m"
        }
      },
                  "hour": {
        "type": "long",
        "meta": {
          "unit": "h"
        }
      },
                  "day": {
        "type": "long",
        "meta": {
          "unit": "d"
        }
      },
      "percent": {
        "type": "long",
        "meta": {
          "unit": "percent"
        }
      },
            "bytes": {
        "type": "long",
        "meta": {
          "unit": "byte"
        }
      }
    }
}

POST my-index-000001/_doc
{
  "nanos" : 1234.5,
  "micros" : 1234.5,
  "ms" : 1234.5,
  "second" : 1234.5,
  "minute" : 1234.5,
  "hour" : 1234.5,
  "day" : 1234.5,
  "percent" : 1234.5,
  "bytes" : 1234.5
}

```
kibana-app-arch automation moved this from To triage to Done in current release Feb 6, 2024
fkanout pushed a commit to fkanout/kibana that referenced this issue Feb 7, 2024
@mattkime @fkanout
[data views] Default field formatters based on field meta values (ela…
fe2651a 
…stic#174973)

## Summary

Default field formatters based on field meta units data.

Note: the smallest unit our formatter will show is milliseconds which
means micro and nanoseconds may round down to zero for smaller values.
elastic#176112


Closes: elastic#82318

Mapping and doc setup for testing - 

```
PUT my-index-000001

PUT my-index-000001/_mapping
{
    "properties": {
            "nanos": {
        "type": "long",
        "meta": {
          "unit": "nanos"
        }
      },
            "micros": {
        "type": "long",
        "meta": {
          "unit": "micros"
        }
      },
      "ms": {
        "type": "long",
        "meta": {
          "unit": "ms"
        }
      },
                  "second": {
        "type": "long",
        "meta": {
          "unit": "s"
        }
      },
                  "minute": {
        "type": "long",
        "meta": {
          "unit": "m"
        }
      },
                  "hour": {
        "type": "long",
        "meta": {
          "unit": "h"
        }
      },
                  "day": {
        "type": "long",
        "meta": {
          "unit": "d"
        }
      },
      "percent": {
        "type": "long",
        "meta": {
          "unit": "percent"
        }
      },
            "bytes": {
        "type": "long",
        "meta": {
          "unit": "byte"
        }
      }
    }
}

POST my-index-000001/_doc
{
  "nanos" : 1234.5,
  "micros" : 1234.5,
  "ms" : 1234.5,
  "second" : 1234.5,
  "minute" : 1234.5,
  "hour" : 1234.5,
  "day" : 1234.5,
  "percent" : 1234.5,
  "bytes" : 1234.5
}

```
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@mattkime @CoenWarmer
[data views] Default field formatters based on field meta values (ela…
43007fd 
…stic#174973)

## Summary

Default field formatters based on field meta units data.

Note: the smallest unit our formatter will show is milliseconds which
means micro and nanoseconds may round down to zero for smaller values.
elastic#176112


Closes: elastic#82318

Mapping and doc setup for testing - 

```
PUT my-index-000001

PUT my-index-000001/_mapping
{
    "properties": {
            "nanos": {
        "type": "long",
        "meta": {
          "unit": "nanos"
        }
      },
            "micros": {
        "type": "long",
        "meta": {
          "unit": "micros"
        }
      },
      "ms": {
        "type": "long",
        "meta": {
          "unit": "ms"
        }
      },
                  "second": {
        "type": "long",
        "meta": {
          "unit": "s"
        }
      },
                  "minute": {
        "type": "long",
        "meta": {
          "unit": "m"
        }
      },
                  "hour": {
        "type": "long",
        "meta": {
          "unit": "h"
        }
      },
                  "day": {
        "type": "long",
        "meta": {
          "unit": "d"
        }
      },
      "percent": {
        "type": "long",
        "meta": {
          "unit": "percent"
        }
      },
            "bytes": {
        "type": "long",
        "meta": {
          "unit": "byte"
        }
      }
    }
}

POST my-index-000001/_doc
{
  "nanos" : 1234.5,
  "micros" : 1234.5,
  "ms" : 1234.5,
  "second" : 1234.5,
  "minute" : 1234.5,
  "hour" : 1234.5,
  "day" : 1234.5,
  "percent" : 1234.5,
  "bytes" : 1234.5
}

```
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this issue Feb 15, 2024
@mattkime @CoenWarmer
[data views] Default field formatters based on field meta values (ela…
f27099f 
…stic#174973)

## Summary

Default field formatters based on field meta units data.

Note: the smallest unit our formatter will show is milliseconds which
means micro and nanoseconds may round down to zero for smaller values.
elastic#176112


Closes: elastic#82318

Mapping and doc setup for testing - 

```
PUT my-index-000001

PUT my-index-000001/_mapping
{
    "properties": {
            "nanos": {
        "type": "long",
        "meta": {
          "unit": "nanos"
        }
      },
            "micros": {
        "type": "long",
        "meta": {
          "unit": "micros"
        }
      },
      "ms": {
        "type": "long",
        "meta": {
          "unit": "ms"
        }
      },
                  "second": {
        "type": "long",
        "meta": {
          "unit": "s"
        }
      },
                  "minute": {
        "type": "long",
        "meta": {
          "unit": "m"
        }
      },
                  "hour": {
        "type": "long",
        "meta": {
          "unit": "h"
        }
      },
                  "day": {
        "type": "long",
        "meta": {
          "unit": "d"
        }
      },
      "percent": {
        "type": "long",
        "meta": {
          "unit": "percent"
        }
      },
            "bytes": {
        "type": "long",
        "meta": {
          "unit": "byte"
        }
      }
    }
}

POST my-index-000001/_doc
{
  "nanos" : 1234.5,
  "micros" : 1234.5,
  "ms" : 1234.5,
  "second" : 1234.5,
  "minute" : 1234.5,
  "hour" : 1234.5,
  "day" : 1234.5,
  "percent" : 1234.5,
  "bytes" : 1234.5
}

```
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@mattkime @fkanout
[data views] Default field formatters based on field meta values (ela…
6a2bf7d 
…stic#174973)

## Summary

Default field formatters based on field meta units data.

Note: the smallest unit our formatter will show is milliseconds which
means micro and nanoseconds may round down to zero for smaller values.
elastic#176112


Closes: elastic#82318

Mapping and doc setup for testing - 

```
PUT my-index-000001

PUT my-index-000001/_mapping
{
    "properties": {
            "nanos": {
        "type": "long",
        "meta": {
          "unit": "nanos"
        }
      },
            "micros": {
        "type": "long",
        "meta": {
          "unit": "micros"
        }
      },
      "ms": {
        "type": "long",
        "meta": {
          "unit": "ms"
        }
      },
                  "second": {
        "type": "long",
        "meta": {
          "unit": "s"
        }
      },
                  "minute": {
        "type": "long",
        "meta": {
          "unit": "m"
        }
      },
                  "hour": {
        "type": "long",
        "meta": {
          "unit": "h"
        }
      },
                  "day": {
        "type": "long",
        "meta": {
          "unit": "d"
        }
      },
      "percent": {
        "type": "long",
        "meta": {
          "unit": "percent"
        }
      },
            "bytes": {
        "type": "long",
        "meta": {
          "unit": "byte"
        }
      }
    }
}

POST my-index-000001/_doc
{
  "nanos" : 1234.5,
  "micros" : 1234.5,
  "ms" : 1234.5,
  "second" : 1234.5,
  "minute" : 1234.5,
  "hour" : 1234.5,
  "day" : 1234.5,
  "percent" : 1234.5,
  "bytes" : 1234.5
}

```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New value added to drive a business result Feature:Data Views Data Views code and UI - index patterns before 8.0 impact:low Addressing this issue will have a low level of impact on the quality/strength of our product. loe:small Small Level of Effort papercut Small "burr" in the product that we should fix. Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph)
Projects
kibana-app-arch
  
Done in current release
Discover
Inbox
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[data views] Default field formatters based on field meta values mattkime/kibana
8 participants
@mattkime @kertal @wylieconlon @dej611 @felixbarny @elasticmachine @davismcphee @petrklapka