[Meta] Field capabilities improvements

[thomasneirynck]

Data-views (and associated utility clients) are the main client to retrieve field-information for index-patterns.

Given that the number of fields can be quite large, and the frequency with which these functions are called, there is a big motivation to improve how and when they're being called.

Low-level improvements:

• use stream based parsing of field-docs.
  • Given that responses can be quite large (order of 10s of megabytes), using JSON.parse to construct a JS-object can cause UX-locking. Instead, consider using stream-based parsing.
  • A similar approach has been taken for file-upload (e.g. csv or geojson) to handle large documents.
• cache field-caps responses, so they are only fetched once to support multi-user
  • (field-level security may complicate this, but this is an opt-in feature, and not the prime use-case)

Data-view API refinement:

• Add field-level filtering
  • the current APIs fetches all fields. A lot of the usage of DataViews does not require all fields. An example would be a Lens-visualization configured to only fetch data based on a few fields.
  • IDataView should provide methods to load partial field-lists. This would use the fields-param from https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html
  • This can be especially important in Dashboard use-cases, because there the user is not presented with field-lists at all. They just expect the chart to work.

UX polish

• Improve the perceived performance in client apps by not blocking the bootstrap of the entire UX until field-list is available
  • (use of field-list building block may alleviate the need for this)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[mattkime]

Closing this issue since DataViewLazy has been implemented here - #173948

The main item not addressed is using stream based parsing of field-docs. Its unclear that would be beneficial as the main field caps slowdown is the collation of responses from different nodes and clusters - and this is nicely improved by requesting fewer fields. Anyway, feel free to reopen or build off this issue if you see benefit.