Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Transforms: Improve data view checks. #181892

Merged
merged 12 commits into from
May 13, 2024

Conversation

walterra
Copy link
Contributor

@walterra walterra commented Apr 26, 2024

Summary

Part of #181603.

For some of the actions in the transform list we need to identify if there's a data view for the target index. The way we identified this was quite inefficient. We had poor caching in place and fetched info for all data views including fields — this can be quite expensive queries!

This update fixes the approach and switches to using dataViews.getIdsWithTitle() in combination with useQuery() to bring the caching in line with how we load and refresh the rest of the transform list.

Before: Lots of field caps requests!

image

After: We do just 1 _search request that gets the data view ids/titles:

image

Checklist

@walterra walterra self-assigned this Apr 26, 2024
@walterra walterra force-pushed the ml-transform-fix-data-view-checks branch from 77b0150 to 2e6abe5 Compare May 2, 2024 09:03
@walterra walterra marked this pull request as ready for review May 3, 2024 11:22
@walterra walterra requested a review from a team as a code owner May 3, 2024 11:22
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@walterra walterra changed the title [ML] Transforms: Fix data view checks. [ML] Transforms: Improve data view checks. May 3, 2024
const dataViewTitle = Array.isArray(item.config.source.index)
? item.config.source.index.join(',')
: item.config.source.index;
const dataViewId = getDataViewIdByTitle(dataViewTitle);
const dataViewListItem = dataViewListItems.find((d) => d.title === dataViewTitle);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each of the uses of dataViewListItems runs a find, these won't be too expensive, but it might be slightly more efficient if useGetDataViewIdsWithTitle returned a map of data views, keyed on title

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Updated the hook to return a map, also renamed everything to be in line with the new returned structure in 9f23cb8.

const toastNotifications = useToastNotifications();

const { getDataViewIdByTitle, loadDataViews } = useSearchItems(undefined);

const { data: dataViewsTitleIdMap } = useGetDataViewsTitleIdMap();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Total nit pick. but rather than having to rename data -> dataViewsTitleIdMap on every use, could it be renamed when being returned from useGetDataViewsTitleIdMap ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hook wraps useQuery so that whole returned object comes from it and data is part of its standard API (like isLoading, error etc.).

Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested, LGTM.
Added a nit pick comment..

Copy link
Contributor

@darnautov darnautov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
transform 441 442 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
transform 393.6KB 393.4KB -267.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
transform 18.8KB 18.8KB +70.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
transform 32 31 -1

Total ESLint disabled count

id before after diff
transform 36 35 -1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @walterra

@walterra walterra merged commit f2945cf into elastic:main May 13, 2024
17 checks passed
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label May 13, 2024
@walterra walterra deleted the ml-transform-fix-data-view-checks branch May 13, 2024 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:skip This commit does not require backporting Feature:Transforms ML transforms :ml release_note:fix v8.15.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants