Limit index pattern field list based on a rule

[tbragin]

The idea behind this came from the beats team. Right now in Metricbeat and in upcoming Filebeat modules they drop the data in a single Elasticsearch index to make sure not too many indices get created. They namespace the fields like module_name.blah.blah.blah. However, for dashboard creation it would be nice to create a "view" on this index that is limited to fields matching pattern "module_name.*" so that metric selection is more sane.

cc: @spalger @Bargs Thoughts?

cc: @tsg @ruflin @monicasarbu

[Bargs]

Is the main pain point the field select dropdown when building visualizations? I wonder if we could improve the experience simply by enhancing that dropdown to make it searchable.

[spalger]

I fully support giving users more control over the shape of index patterns, but I am a little bit worried about one thing:

I think that right now Kibana communicates/implies that there is a direct correlation between Elasticsearch indices and Kibana index patterns. When we start altering the representation of the index in Kibana, and only Kibana, that correlation starts to break down. Kibana starts to misrepresent elasticsearch indexes.

This is a concern that started to surface in #7402 and I think becomes more of an issue with a change like this.

Do others feel the same way?

[spalger]

Also, its probably fair to say that this is blocked by #3351

[tsg]

I agree that a limiting "view" is probably more than we actually need for a good user experience. For sure #2130 will help a lot already.

In addition, I'd prefer to work towards a solution where we can create smaller index patterns to start with. The reason our index patterns are huge is that they contain all the fields from all the modules. If we would include only the fields from the modules that are actually enabled, then the number of fields would reasonable. We could almost do that today, i.e. when Metricbeat starts, generate the index pattern for Kibana with the enabled modules only and index it to .kibana. But there would be issues if you run two Metricbeat instances with a different set of modules enabled. They would overwrite each other's fields, because there's only one index pattern doc in kibana. If we try to read and merge the doc on the Beats side, there's going to be race conditions.

This could be solvable if Kibana would have some sort of API to "add fields" to an existing index pattern, overwriting duplicate fields. Since we anyway want to have a better way of adding our dashboards/searches/patterns, I think this is an idea worth exploring.