Fix KQL value suggestions for nested fields

[Bargs]

Summary

Fixes #54642

In #51774 I flagged nested fields as non-aggregatable in the index pattern so that they would not show up in the Visualize editor until we have implemented nested field support for aggregations. This had the unintended effect of turning off value suggestions for nested fields in KQL because we don't retrieve suggestions for non-aggregatable fields. So in hindsight marking nested fields as non-aggregatable was too broad of a change. I've reverted that change and instead I'm now filtering out nested fields from Visualize at the agg param level.

I really think we should get this into 7.6 because if we don't, users will have to manually refresh their index patterns in 7.7 to fix the value suggestion bug.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

- [] This was checked for cross-browser compatibility, including a check against IE11
- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
- [ ] Documentation was added for features that require explanation or tutorials

• Unit or functional tests were updated or added to match the most common scenarios
  - [ ] This was checked for keyboard-only and screenreader accessibility

For maintainers

- [ ] This was checked for breaking API changes and was labeled appropriately

• This includes a feature addition or change that requires a release note and was labeled appropriately

[timroes]

While I agree that it's stupid, if the user need to refresh their index pattern in general, if we implement aggregations on nested fields, we are lacking a couple of places here where we manually need to exclude nested fields for now:

• Lens
• TSVB
• Maps
• Canvas
• ???

I am not sure if there are more then that, but currently couldn't come up with any other.

Started to look into Lens, and there are still a couple of type mismatches, while not every of this places might be trivial to fix. So we basically have a couple of options now (and none makes me really happy :D):

1. We mark it as aggregatable right now (as this PR does), and try to limit down all the places manually that would consume the aggregatable flag but don't support nested fields yet. Might cause more work than we expect right now, and I am not 100% certain we'll think about all the places.
2. We keep it as unaggregatable in the index pattern, and just adjust the KQL logic to ignore the flag for nested fields manually, and load auto completion anyway. Means we need a special hack in the auto completion provider + the issue, that we'll require users with nested fields to refresh their index pattern, once we support nested fields in aggregations.
3. We just ignore autocompletion on nested field values for now and take it as a limitation we know. Will not require any dirty hacks in the auto completion provider, but will have the same issue still, users will need to refresh their index pattern once we support aggregating over nested fields.

None of those solutions make me really happy. From a user perspective I think (1) would be the best, but I am a bit afraid given the amount of changes we still need to make to different apps for this, given our release time line.

Update: During fixing some of those places it seems that not only switching the aggregatable flag for the child nodes is causing issues in some apps, but also that the fields are marked as sortable, and that the nested type fields exist at all make them selectable in a range of places (and then fail).

[Bargs]

Yeah I agree none of those are ideal. One thing to consider is that prior to #51774 we displayed nested fields in all these unsupported places for years, probably back to 4.x at least. So it's an annoyance, but probably one most nested field users are already used to. Nested field support in KQL is new, so shipping a bug with it that makes it look slightly broken may leave a bad initial impression.

So to add to your list of options, we could also just revert #51774 and leave things as they have been.

Actually, as I was typing this, I also realized that when we add nested support to Visualize we will need to mark these fields as aggregatable, and then we will start showing nested fields in all the other unsupported places again, so I think marking them as unaggregatable now is not a good idea.

[Bargs]

@timroes I reverted your last TSVB commit so it can be made in a separate PR like we talked about. Let me know if there's anything else you think we need in this PR before merging.

[timroes]

Tested on Chrome Linux, works as expected. Will create a followup PR disabled nested fields in more apps/places for selection.

[timroes]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 9660f1c

History

• 💚 Build #21253 succeeded 53e8072
• 💚 Build #20712 succeeded f7b74c0
• 💚 Build #20509 succeeded afcaa34
• 💚 Build #20332 succeeded 7044931

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream