[Lens] Paginate index pattern list, especially if there are > 1000 index patterns stored

[wylieconlon]

No description provided.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[stratoula]

This is a unified search issue but I am not sure about how the dataview service works right now. Is the above feasible @elastic/kibana-data-discovery ?

[kertal]

correct me if I'm wrong @mattkime , I think incremental loading is not built in. However, did we render the whole list 2020? because if so, in the meantime, we don't, right?

[mattkime]

Does the UI handle a large number of data views well?

As far as the service side of things, the logic around this isn't terribly clever but perhaps its fine. Might be worth a performance test to make sure it scales. We load a small set of attributes for the first 10k data views - https://github.com/elastic/kibana/blob/main/src/plugins/data_views/common/data_views/data_views.ts#L366

[stratoula]

The UI fetches them with the getIdsWithTitle method and then uses an EUI selectable to render them. So nothing sophisticated here. About the EUI selectable performance I am not sure tbh.

  useEffect(() => {
    const fetchDataViews = async () => {
      const savedDataViewRefs: DataViewListItemEnhanced[] = savedDataViews
        ? savedDataViews
        : await data.dataViews.getIdsWithTitle();
      // not propagate the adHoc dataviews on the list for text based languages
      const adHocDataViewRefs: DataViewListItemEnhanced[] =
        (!isTextBasedLangSelected && adHocDataViews?.map(mapAdHocDataView)) || [];

      setDataViewsList(savedDataViewRefs.concat(adHocDataViewRefs));
    };
    fetchDataViews();
  }, [data, currentDataViewId, adHocDataViews, savedDataViews, isTextBasedLangSelected]);

A performance test would be a good idea yes. What are you using for testing the performance of dataviews? I mean do you have a tool which produces plenty of dataviews or something like that to help with the testing?

I think it would be a good idea to add pagination both in the api and the client.

[kertal]

2 cents from my end. The EuiSelectable is using virtualization for those lists. So there shouldn't be a performance issue, unless the preparation of the list is expensive, but this doesn't seem to be the case. Of course loading all entries at once are taking memory, however, our virtualized field list is dealing with 50k fields, without a rendering issue ... hasn't always been the case.

this issue was created in 2020, I think back then it worked differently, maybe just a unvirtualized drop down:

[stratoula]

is using virtualization for those lists

You are right! So this should not be an issue.

this issue was created in 2020, I think back then it worked differently, maybe just a unvirtualized drop down:

@kertal honestly I don't know 😂 I think that it was using the EUISelectable always but I am not sure if it was virtualized back then. 🤔

[mattkime]

Seems we could at least at a performance test when working with a high number of data views (over 1000)