[Lens] hide missing fields errors until the data view has some fields

[drewdaemon]

Describe the feature:
When there is no data in a data view, we show a slew of missing field errors. Technically, this is correct. Since they don't have data, no fields are present, so they're all missing.

However, in this case we can provide more useful messaging that doesn't look as severe. We can be pretty sure that this is more of a no-data case, than a broken visualization.

Describe a specific use case for the feature:
In #143673 we stopped blocking visualization render when there are missing fields. This improved the pre-ingest integration dashboard look by making the missing field errors less in-your-face.

However, we are still showing the user errors when they haven't done anything wrong. They're just waiting for data.

cc: @ruflin @MichaelMarcialis

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[drewdaemon]

Relates to #149836

[ruflin]

I really like that we know which of the fields are missing. This is not only helpful to users of the visualisations but especially to the ones building the visualisations. It would be nice if we could show it much more subtle somehow.

[drewdaemon]

@ruflin thank you for the feedback.

This is not only helpful to users of the visualisations but especially to the ones building the visualisations.

Do the visualization authors ever build their visualizations without first ingesting some data? If not, this change shouldn't impact their experience.

This issue is about hiding the missing fields list before ingest (i.e. when a user has just installed integration assets but is waiting on data). Before ingest, all fields are counted missing through no fault of the user, so we're considering treating it as a no-data state.

As soon as the first document is indexed, we would show the missing field errors (if there are any) as we do today.

WDYT?

[ruflin]

Do the visualization authors ever build their visualizations without first ingesting some data? If not, this change shouldn't impact their experience.

I'm more thinking about the debugging use case. Vis built, exported, someone else continues to work on it and for some reason, it doesn't show results. Maybe ingestion field changed, ingest pipeline was modified and had unexpected side effects, ...

As soon as the first document is indexed, we would show the missing field errors (if there are any) as we do today.

There are quite a few dataset where certain fields only show up when it happened for the first time. Lets take nginx logs as example. As long as no request came in containing a source.address, the field will not be populated. Or in the Elasticsearch case, as long as there is no exception happening in the log file, I think a few less fields are populated. I guess the same is true for Kibana logs. And this can also happen in the metric case as some metrics are not populated at first (or never if something never happens).

I hope this answers the question indirectly.

[drewdaemon]

Vis built, exported, someone else continues to work on it and for some reason, it doesn't show results. Maybe ingestion field changed, ingest pipeline was modified and had unexpected side effects, ...

I think this might still be covered here because the suggested behavior is

• if there are no fields, show a non-error message saying there is no data
• if there are some fields, but some are missing, show errors for the missing fields like we do now

So in either case, the viewer would be warned that something is wrong/changed.

There are quite a few dataset where certain fields only show up when it happened for the first time.

Hmmm, this is an important point. You're saying that not all fields are necessarily included in the first document that is indexed, right?

I think the key question is: does the field caps API report these fields even though none of the documents contain values for them?

If yes — the data view will still contain these fields so Lens will not detect them as missing ✅.
If no — the data view will not, and missing field errors would start showing up as soon as the first document is indexed, even though having those fields be missing isn't necessarily a problem which, as you're saying, calls into question the "error" classification.

Since the indexes are created from index templates which contain all the fields in their mappings, I'm thinking the answer is "yes." But, I should validate this.

[ruflin]

Hmmm, this is an important point. You're saying that not all fields are necessarily included in the first document that is indexed, right?

Correct.

I think the key question is: does the field caps API report these fields even though none of the documents contain values for them?

The answer is currently yes. But it is something I have been challenging for some time and especially for ECS fields (where we load too many) we move to dynamic templates to not have this happening. Ideally the field caps API would tell you what fields are actually used (have content) and what is only template (unfortunately in our case the mapping too).

[drewdaemon]

I think the key question is: does the field caps API report these fields even though none of the documents contain values for them?

The answer is currently yes. But it is something I have been challenging for some time and especially for ECS fields (where we load too many) we move to dynamic templates to not have this happening.

So, it seems to me like the change described in this issue does improve the current situation (?), but may be disrupted by this change. Do you have an idea on timeline?

Ideally the field caps API would tell you what fields are actually used (have content) and what is only template (unfortunately in our case the mapping too).

I think this would be nice because it would allow us to discriminate between "hard" missing and "soft" missing cases in the UI. This change would certainly mean (modest?) changes for the data view system (cc: @mattkime ).

[mattkime]

I think the key question is: does the field caps API report these fields even though none of the documents contain values for them?

Yes, there are two similar questions - "What are all the fields?" and "What are all the fields for a given filter?" Kibana as a whole could be a bit more sophisticated as to how it treats these two cases although some apps (lens and discover, I think) do a pretty good job. IMO fields with data in the current context should be prominent but all fields should be available in some manner.

@drewdaemon Perhaps we should talk to someone on the ES team who knows how field caps determines if a given field has values. I know this functionality is there but I'd like to understand it in better detail.

[ruflin]

I think is is related / overlaps with #24709

[flash1293]

My take on this:

The way we set up the "managed" data collection (via integrations/fleet/elastic-agent), this case is common and will get more common, which makes this issue increasingly more important. Fields will not show up in the mapping until data got actually sent by the shippers, but it's not a binary question - each field can show up or not show up at any point in time (depending on the configuration of the shipper, some fields might never show up because certain parts of the data aren't collected).

As fields are defined in a very dynamic manner by their suffix (e.g. *.ip is mapped as ip), it's not very meaningful to look for "hypothetical" fields that could exist in the mapping given the right data, as almost everything could and it's not a guarantee we will ever see data for it.

On the flip side, the case where a field missing in the mapping is a serious issue that needs admin attention might be true in some situations, but is not how most setups are configured.

Things we could do:

• Make this a special case for managed dashboards - in case of managed dashboards, let the integration author decide how they want to treat missing field errors (e.g. hide them)
• Make this a special case for the logs-* and metrics-* data streams as the mappings in these are set up in this dynamic manner by default
• Make this a special case by the dynamic configuration of the indices we are hitting - if new fields can be added by incoming data, treat missing fields as missing data (no results), if the mapping is set to strict, treat missing fields as config error (current behavior)
• Make this the default for all cases (like some competitors do)

What do you think, @markov00 @ruflin ?

[ruflin]

I like option 1 and 2 with preference on 1. For o11y and security use case, I don't see a problem with 4 but it's a breaking change I don't think it is worth to go through it. Potentially 4 could be done on a project level. I think of it as a hierarchy of settings: Project < Dashboard < Visualisation. Dashboard seems to be a good middleground to start.

[flash1293]

An advantage of 2 is that it avoids another bit of explicit configuration that needs to be managed.

[dej611]

I like option 3 better vs others, but it might take a bit to get it up and running.
Maybe option 2 can be a quick win, with the option to extend it to option 3 in the medium/longer term.

[markov00]

I agree with @dej611 option 3 can probably catch the issue at the source (dynamic mappings mean that some fields can be missing), but 2 is a quick with without adding more complexity to the lens/dashboard configuration.