Standardize filter management mechanisms

[asteriscos]

Description

We need to refactor and standardize the filtering mechanisms so all the embeddable dashboards and reports can make use of the filter state without using any of the legacy methods. This is a step forward towards deprecating the Discover legacy, AngularJS, and also integrating the new embeddable dashboards with the PDF reports.

Functional requirements

• The filter system must consider the search bar filters
• The filter system must consider the hideManager alerts configuration
• The filter system must consider if an agent is pinned
• The filter system must consider the allowed agents by RBAC
• The filter system must consider the selected cluster

[Machi3mfl]

Scope

The aim is to generate a standard solution that allows each system module (such as vulnerabilities) to manage its own data source.
It is important that the solution is standard and allows:

• Retrieving filters (getFilters)
• Fetching data (fetchData)

The goal is for each module to be able to manage its own data source, to be able to decouple, decentralize, and give responsibility to each module for its own data source.

Problematic

Currently, the state of the current filters comes from different sources:

• FilterManager (OSD Filter Manager)
• Session Storage (Browser)

Every module has specific default filters, and need to preserve and merge them with the user filters if are inside the module.

Use Case

• When we navigate to specific modules, exists a filter that is applied by default. Also, we need to check the OSD filter manager to see if the filters correspond with the current module.

Plan

• Research the use of filters using the vulnerabilities detector module (Add cluster filter by default in vulnerabilities dashboard and inventory tab #6399)

• Create a generic solution design

  • Create a data source filter manager with unit tests Standardize filter management mechanisms #6499 (comment)
  • Create a hook (useDataSource) that simplifies the use of data source and unit tests Standardize filter management mechanisms #6499 (comment)

• Implement a solution for the specific module (vulnerability detector) Standardize filter management mechanisms #6499 (comment) (26/03/24)

  • Implement the hook on the Dashboard tab
  • Implement the hook on the Inventory tab
  • Implement the hook on the Events tab

• Add unit tests to the final solution Standardize filter management mechanisms #6499 (comment)

• Add manual test cases in PR Add standard modules filters management #6534 (comment)

•  Hide the close icon button in the search bar when the filters are added by the data source. That filters cannot be removed and are managed by the current module data source (Ex: vulnerabilities)

Screen.Recording.2024-04-03.at.14.17.47.mov

[Machi3mfl]

Create a generic solution design

Data source filter manager

Is a class that is responsible for managing the filters of a data source (Important: the filters are not managed by the data source itself, it's managed by the filter manager). This allows the filters to be managed in a centralized way but at the same time makes it responsive to the data source to define its filters.

Use cases

• The data source filter manager receives a data source and the filters from the filter manager or another source. The filters have the type Filter.

• When the data source filter manager receives the filters:

  • Filter the filters received and only keep the filters that was added by the user.
    Condition: When the filter not have the property meta.controlledBy, $state.isImplicit and the meta.index is not the same as the data source index pattern.

• Then is able to:

  • Retrieve the filters applied by the user and the data source filters (For instance: the vulnerabilities module). Every module has the responsibility to manage its own filters.
  • Fetch the data from the data source using the filters and add extra filters that are not visible to the user.
    For instance: hide manager, pinned agent, allowed agents, etc.

Solution

export type tDataSource = {
    id: string;
    title: string;
    select(): Promise<void>;
    getFilters: () => Promise<tFilter[]> | tFilter[];
    setFilters: (filters: tFilter[]) => Promise<void> | void;
    getFields: () => Promise<any[]> | any[];
    getFixedFilters: () => tFilter[];
    getFetchFilters: () => tFilter[];
    fetch: (params: tSearchParams) => Promise<any>;
    toJSON(): object;
}

export interface IDataSourceFactoryConstructor<T extends tDataSource> {
    new(id: T['id'], title: T['title']): T;
}

export type tDataSourceFactory<T extends object, K extends tDataSource> = {
    create: (DataSourceType: IDataSourceFactoryConstructor<K>, data: T) => Promise<K>;
    createAll: (DataSourceType: IDataSourceFactoryConstructor<K>, data: T[]) => Promise<K[]>;
}

export type tDataSourceFilterManager = {
    fetch: () => Promise<any>;
    setFilters: (filters: tFilter[]) => void;
    getFilters: () => tFilter[];
    getFixedFilters: () => tFilter[];
    getFetchFilters: () => tFilter[];
}

export type tDataSourceRepository<T extends object> = {
    get(id: string): Promise<T>;
    getAll(): Promise<T[]>;
    setDefault(dataSourceData: T): Promise<void> | void;
    getDefault(): Promise<T | null> | T | null;
}

export type tDataSourceSelector<T extends tDataSource> = {
    existsDataSource: (id: string) => Promise<boolean>;
    getFirstValidDataSource: () => Promise<T>;
    getAllDataSources: () => Promise<T[]>;
    getDataSource: (id: string) => Promise<T>;
    getSelectedDataSource: () => Promise<T>;
    selectDataSource: (id: string) => Promise<void>;
}

export type tFilter = Filter & {
    meta?: {
        removable?: boolean;
    }
};

export type tSearchParams = {
    filters?: tFilter[];
    query?: any;
    pagination?: {
        pageIndex?: number;
        pageSize?: number;
    };
    fields?: string[],
    sorting?: {
        columns: {
            id: string;
            direction: 'asc' | 'desc';
        }[];
    };
    dateRange?: {
        from: string;
        to: string;
    };
}

Unit tests

[Machi3mfl]

Create a generic solution design

Data source hook

Create a react hook that:

• simplifies the use of data sources, related filters, fetching data
• reduce the boilerplate

type tUseDataSourceProps<T extends object, K extends PatternDataSource> = {
    DataSource: IDataSourceFactoryConstructor<K>;
    repository: tDataSourceRepository<T>;
    factory?: PatternDataSourceFactory;
    filterManager?: tFilterManager;
    filters?: tFilter[];
}

type tUseDataSourceLoadedReturns<K> = {
    isLoading: boolean;
    dataSource: K;
    filters: tFilter[];
    fetchFilters: tFilter[];
    fixedFilters: tFilter[];
    fetchData: (params: Omit<tSearchParams, 'filters'>) => Promise<any>;
    setFilters: (filters: tFilter[]) => void;
    filterManager: PatternDataSourceFilterManager;
}

type tUseDataSourceNotLoadedReturns = {
    isLoading: boolean;
    dataSource: undefined;
    filters: [];
    fetchFilters: [];
    fixedFilters: [];
    fetchData: (params: Omit<tSearchParams, 'filters'>) => Promise<any>;
    setFilters: (filters: tFilter[]) => void;
    filterManager: null;
}

Unit tests

[Machi3mfl]

Implemented on Vulnerability Detection Module

Screen.Recording.2024-03-27.at.16.55.26.mov

This solution uses the new:

• Data Source solution
• useDataSource hook
• Wazuh discover with useDataSource hook added

[Machi3mfl]

Add unit tests to the final solution

[Machi3mfl]

How to use the new data source solution

(Coming soon ⏳)