Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][POC] Migrate Sourcerer component to Ad-hoc data views #138181

Open
YulNaumenko opened this issue Aug 4, 2022 · 0 comments
Open
Labels
Team:Detection Engine Security Solution Detection Engine Area technical debt Improvement of the software architecture and operational architecture

Comments

@YulNaumenko
Copy link
Contributor

YulNaumenko commented Aug 4, 2022

Currently all the data view management in security_soluition is organized with Sourcerer component and sourcerer API

Problems we are trying to solve by migrating to Ad-hoc Data Views:

  1. Resolve performance bottleneck related to uncached requests for ES fieldsCaps to check the indices availability.
  2. Inability to define the page persistent data view with index patterns, which are used by the page components (network, hosts, user pages, etc). Documented GH issue. This cause another performance decreasing by fetching unnecessary data.
  3. Minimize the data amount for the index fields fetching.
  4. It currently includes the alerts index, which means users can't use this data view in rule creation without creating alerts on alerts
  5. Separating out the alerts index would remove all the polling logic sourcerer currently has

Potencial blockers:

  1. Ability to create the predefined data views (persistent and in-memory?) for security_solution without the user interaction. This logic also have the dependency to the current integrations state. If there is no integrations we shouldn't create data views and create it right after the user added security integration.
  2. Ability to show in UI the disabled index aliases which are default for security but doesn't have indices yet.
    Screen Shot 2022-08-04 at 1 42 40 PM
  3. Ability to show the index field description in the FieldBrowser component. Currently we are reading all that information from the file with BeatFields constant, which is updated manually.
  4. Make data view readonly, to pre-event adding prohibited indices for the Detections pages:
    Screen Shot 2022-08-04 at 2 52 49 PM
  5. Check with UX if this will be a similar user experience when we replace checkbox "Show only detection alerts" for the timelines page with the proper (in-memory?) Ad-hoc data view?
  6. Manage the legacy data views, with index patterns which was excluded from the default data view in 8.0.

Potential migration steps:

  1. Investigate a possibility to replace sourcerer API with the direct dataView plugin API usage and Ad-hoc data views.
  2. Replace existing Sourcerer UI component with Ad hoc data views.
  3. Simplify the data flows across the components which is using sourcerer, keep the minimum number of React hooks responsible for create/update and read state from the store. Could be done independently of the ad-hoc data views work.
@YulNaumenko YulNaumenko added technical debt Improvement of the software architecture and operational architecture Team:Security Solution Platform Security Solution Platform Team labels Aug 4, 2022
dhurley14 added a commit that referenced this issue Apr 25, 2023
…ch strategy with api provided by kibana data views plugin (#149360)

## Summary

Original outline: #138181

Issues outlining the objective of this pr:
#142903 and
#142907


#### Overview

Since the data views plugin was introduced, maintaining our own apis for
fetching sourcerer saved objects (data views) and additional types has
become cumbersome and inefficient. The data views plugin provides both
an efficient caching of data view saved objects and a unified interface
for creating ad-hoc data views (see the changes to the `useFetchIndex`
hook in this PR) so that our code can now rely on a single type of saved
object to interface with when fetching data.

This PR is another step towards replacing sourcerer with the data view
picker provided by kibana platform (which benefits users by maintaining
consistency around data source selection UX) and additionally provides
benefits to developers in the security solution by allowing us to reduce
state-management complexity in components that rely on old
`indexPattern` types or data view types.

---------

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
@yctercero yctercero added Team:Detection Engine Security Solution Detection Engine Area and removed Team:Security Solution Platform Security Solution Platform Team labels May 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Detection Engine Security Solution Detection Engine Area technical debt Improvement of the software architecture and operational architecture
Projects
None yet
Development

No branches or pull requests

2 participants