[Discover] [Reporting] Discover and CSV export handle conflicting object and flattened field leaf values differently

[davismcphee]

Summary

This issue arises when dealing with a data view that has a conflicting field of types object and flattened.

Normally in Discover, it's only possible to add the entire flattened field as a column, but not its properties since they aren't indexed as separate fields:

But it's the opposite for object fields, where each property is flattened and indexed separately, so only the leaf values can be added as columns:

When there's a conflict like this, Discover doesn't know which behaviour is correct, so it highlights the field as conflicting in the field list and allows adding both the root and leaf values (which it labels as multi fields) as columns:

But the way it handles them in the grid is different between documents. For the document where user is a flattened field, it displays the full object for the root value column and empty for the leaf value columns. For the document where user is an object field, it displays empty for the root value column and the property values for leaf value columns:

Even more confusing is that when exporting a CSV with the user.first and user.last columns in the table, the CSV includes values for both the document with the flattened field and the document with the object field:

"user.first","user.last"
"John, Alice","Smith, White"
"Bob, Foo","Joe, Bar"

Solution

It's not clear which behaviour is correct or incorrect in this situation or how to resolve it, but there seem to be a few options:

• Change the handling of conflicting fields in Discover so users can't end up in this situation. Does it make sense to allow adding both the root and leaf values when this type of conflict occurs (since it may be misleading to users), and should the leaf values be labelled as multi fields?
• Align the Discover behaviour with the CSV export. This would resolve the inconsistency, but if we go this route we should make sure flattened fields in general support adding leaf values as columns, otherwise the conflicting field type would have better support for leaf values than flattened fields directly.
• Align the CSV export with the Discover behaviour. This would basically remove some functionality from the CSV export, but at least the behaviour would be consistent with Discover.

Reproduction steps

The above scenario can be reproduced by running the following commands in console, creating a flat-conflict-* data view, and navigating to Discover:

PUT flat-conflict-1
{
  "mappings": {
    "properties": {
      "user": {
        "type": "flattened" 
      }
    }
  }
}

PUT flat-conflict-1/_doc/1
{
  "group" : "fans",
  "user" : [
    {
      "first" : "John",
      "last" :  "Smith"
    },
    {
      "first" : "Alice",
      "last" :  "White"
    }
  ]
}

PUT flat-conflict-2
{
  "mappings": {
    "properties": {
      "user": {
        "type": "object" 
      }
    }
  }
}

PUT flat-conflict-2/_doc/1
{
  "group" : "fans",
  "user" : [
    {
      "first" : "Bob",
      "last" :  "Joe"
    },
    {
      "first" : "Foo",
      "last" :  "Bar"
    }
  ]
}

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Align the Discover behaviour with the CSV export. This would resolve the inconsistency, but if we go this route we should make sure flattened fields in general support adding leaf values as columns, otherwise the conflicting field type would have better support for leaf values than flattened fields directly.

Thx for raising this, I think your 2nd suggestion would be the best one in terms of UX. I've been working with flattened fields recently, and felt the need to select leaf nodes. However in terms of effort I guess it's not easy to do, since those leaf nodes are not returned by field_caps.
What would be possible is to analyze the returned result, extract flattened fields subfields, and add those to a new subfield section of the field popover, so it's selectable. However this sounds simple, but ain't simple high likely.

But I would like this functionality!

[kertal]

Or we just add flattened fields support to ES|QL :)

[drewdaemon]

Or we just add flattened fields support to ES|QL :)

Lower-hanging fruit than nested fields I believe :)

[kertal]

Or we just add flattened fields support to ES|QL :)

Lower-hanging fruit than nested fields I believe :)

@drewdaemon where's the code that we can hack it in? :)

[drewdaemon]

Low hanging... but possibly still out of my reach haha. But hey, if we submit a PR, I'm sure the team would be happy to review :)