[data grid] multiple data sets state management & filtering

[davesagraf]

The problem in depth

Hi everyone!

I would like to share some struggles I've been having with MUI X Data Grid Pro recently and, hopefully, someone much more experienced than I could point me in the right direction or, unfortunately, just confirm my not so optimistic suspicions about the current capabilities of the Data Grid.

So, I have a functionality which is something like this:
A user logs into their account and sees a dashboard which has 5 Data Grid components below (only 1 shown at a time).

Each of those components opens in a separate tab (my app is a SPA, so they are toggled via ToggleGroup, also by MUI).
Those tabs show different Data Grid components which are based on assetType (which are 'webService', 'domain', 'ssl', 'ip' and 'port'), but anyway, essentially, we have just 5 different components (each using Data Grid), shown separately (one at a time).

Each table also has 5 tabs of their own, 4 of which are based on the asset status filter.
Initial data is without a status filter and is shown on the all tab. The rest are corresponding to statuses new, inProgress, fixed and accepted.

The most interesting part comes in inside of each of those components, however.

This is an example of my custom SslTable:

//imports
interface IInfrastructureScanningDomainTableProperties {
  domain: IInfrastructurePageDomain;
  tableData: IInfrastructurePageSslTableView[];
}

export const InfrastructurePageSslTable: FunctionComponent<IInfrastructureScanningDomainTableProperties> = observer(
  ({ domain, tableData }) => {
    tableData = tableData ?? [];

    // make data copy to separate rows views & counters filtered by statuses logic
    const rows = [...tableData];

    const apiRef = useGridApiRef();
    const theme = useTheme();

    const [filterButtonEl, setFilterButtonEl] = React.useState<HTMLButtonElement | null>(null);
    const [selectedRows, setSelectedRows] = React.useState<GridRowId[]>([]);

    const isHaveResults = tableData.length !== 0;

    // filter data copy and NOT the original data
    const filteredData = useMemo(() => {
      const memoizedFilteredData = rows.filter(
      (entity) => entity.verificationStatus === domain.ui.sslTableView.entity.filter || domain.ui.sslTableView.entity.filter === 'all')
      return memoizedFilteredData;
    }, [rows, domain.ui.sslTableView.entity.filter, domain.ui.sslTableView.entity.textFilter]);
    
    const columns = useMemo(() => [
      //...rest of the columns,
      InfrastructurePageCommonVerificationStatusColumn(domain, '80px'),
    ], []);

    const filterColumns = ({ field, columns, currentFilters }: FilterColumnsArgs) => {
      const filteredFields = currentFilters?.map((item) => item.field);
      return columns
        .filter((colDef) => colDef.filterable && (colDef.field === field || !filteredFields.includes(colDef.field)))
        .map((column) => column.field);
    };

    const getColumnForNewFilter = ({ currentFilters, columns }: GetColumnForNewFilterArgs) => {
      if (currentFilters.length >= 5) {
        return null;
      }
      const filteredFields = currentFilters?.map(({ field }) => field);
      const columnForNewFilter = columns
        .filter((colDef) => colDef.filterable && !filteredFields.includes(colDef.field))
        .find((colDef) => colDef.filterOperators?.length);
      return columnForNewFilter?.field ?? null;
    };

    const handlePaginationModelChange = () => {
      const dashboardElement = document.getElementById("sticky-area-container");
      if (dashboardElement) {
        setTimeout(() => dashboardElement.scrollIntoView({ block: 'end' }), 100)
      }
    };

    // DO NOT CHANGE params & dependencies
    const debouncedHandleStateChange = useCallback(debounce(() => {
      domain.ui.handleStateChange(rows, 'ssl', apiRef);
    }, 500), [apiRef]);

    return (
      <Grid xs={12} lg={12}>
        <Box>
          <Box sx={{ minWidth: 0, maxWidth: '700px', position: 'relative' }}>
            <InfrastructurePageFilterToggleGroup assetType={'ssl'} data={tableData} domain={domain} />
            {
              !!selectedRows.length &&
                <Box>
                  <CustomMultiselect selectedRows={selectedRows} domain={domain} />
                </Box>
            }
          </Box>
          <Box sx={{ display: 'flex', minWidth: 0, width: '323px', gap: '8px', marginLeft: 'auto' }}/>
        </Box>
        <Grid mt={'-12px'} sx={{ minWidth: 0, width: '1328px', height: '60vh' }}>
          {isHaveResults && (
            <DataGridPro
              apiRef={apiRef}
              // USE onFilterModelChange and NOT onStateChange!!!
              onFilterModelChange={debouncedHandleStateChange}
              localeText={localeText}
              checkboxSelection
              rows={filteredData}
              columns={columns}
              pagination
              onPaginationModelChange={handlePaginationModelChange}
              onRowSelectionModelChange={setSelectedRows}
              initialState={initialState}
              //rest of Data Grid Pro props
            />
          )}
        </Grid>
        <Grid mt={'10vh'}></Grid>
      </Grid>
    );
  },
);

And this is domain.ui.handleStateChange function logic which is called inside the debouncedHandleStateChange function:

handleStateChange (tableData: any, assetType: string, apiRef: React.MutableRefObject<GridApiPro>): Function {
    let filteredDataRows = tableData;
    
    if (apiRef.current?.state?.filter?.filteredRowsLookup) {
      const dataGridRowIDs = new Set(
        Object.entries(apiRef.current.state.filter.filteredRowsLookup)
          .filter(([_, value]) => value === true)
          .map(([key, _]) => key)
      );
      
      // DO NOT REMOVE!!!
      // asset data & filter counters change for Data Grid onFilterModelChange
      
      const filteredDataRowsByAll = [...tableData.filter((item) => dataGridRowIDs.has(item.id))].filter((item) => this[`${assetType}TableView`].entity.filter === 'all');
      this[`${assetType}TableViewRowsCountByAll`].setValue(filteredDataRowsByAll.length);
      
      const filteredDataRowsByNew = [...tableData.filter((item) => dataGridRowIDs.has(item.id))].filter((item) => item.verificationStatus === ENTITY_VERIFICATION_STATUS.new);
      this[`${assetType}TableViewRowsCountByNew`].setValue(filteredDataRowsByNew.length);
      
      const filteredDataRowsByInProgress = [...tableData.filter((item) => dataGridRowIDs.has(item.id))].filter((item) => item.verificationStatus === ENTITY_VERIFICATION_STATUS.inProgress);
      this[`${assetType}TableViewRowsCountByInProgress`].setValue(filteredDataRowsByInProgress.length);
      
      const filteredDataRowsByFixed = [...tableData.filter((item) => dataGridRowIDs.has(item.id))].filter((item) => item.verificationStatus === ENTITY_VERIFICATION_STATUS.fixed);
      this[`${assetType}TableViewRowsCountByFixed`].setValue(filteredDataRowsByFixed.length);
      
      const filteredDataRowsByAccepted = [...tableData.filter((item) => dataGridRowIDs.has(item.id))].filter((item) => item.verificationStatus === ENTITY_VERIFICATION_STATUS.accepted);
      this[`${assetType}TableViewRowsCountByAccepted`].setValue(filteredDataRowsByAccepted.length);
    }
    
    return filteredDataRows;
}

This is a custom ToggleGroup component which I'm using to display each of the filtered data tabs with corresponding counters:

//imports
interface IInfrastructurePageFilterToggleGroup {
  domain: IInfrastructurePageDomain;
  data:
    | IInfrastructurePageDomainTableView[]
    | IInfrastructurePageIpAddressesTableView[]
    | IInfrastructurePagePortTableView[]
    | IInfrastructurePageSslTableView[];
  assetType: keyof IInfrastructureTableViewFiltersByAssetCategory;
}

export const InfrastructurePageFilterToggleGroup: FunctionComponent<IInfrastructurePageFilterToggleGroup> = observer(
  ({ domain, data, assetType }) => {
    const value = domain.ui[`${assetType}TableView`]?.entity.filter || 'all';

    const filterData = (data: any[], filterBy: InfrastructureAssetVerificationStatuses | 'all') =>
      data.filter(
        (
          entity:
            | IInfrastructurePageWebServiceTableView
            | IInfrastructurePageDomainTableView
            | IInfrastructurePageIpAddressesTableView
            | IInfrastructurePagePortTableView
            | IInfrastructurePageSslTableView
            | IInfrastructurePageResponsiblePersonWithGroupingTableView,
        ) => entity.verificationStatus === filterBy,
      ).length || 0;

    const ignoredCount = filterData(data, ENTITY_VERIFICATION_STATUS.ignore);

    const All = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='All'
        filterBy='all'
        isActive={'all' === value}
        numberOfEntity={domain.ui[`${assetType}TableViewRowsCountByAll`].value}
      />
    )
    
    const New = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='New'
        filterBy={ENTITY_VERIFICATION_STATUS.new}
        isActive={ENTITY_VERIFICATION_STATUS.new === value}
        numberOfEntity={domain.ui[`${assetType}TableViewRowsCountByNew`].value}
      />
    )
    
    const inProgress = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='In Progress'
        filterBy={ENTITY_VERIFICATION_STATUS.inProgress}
        isActive={ENTITY_VERIFICATION_STATUS.inProgress === value}
        numberOfEntity={domain.ui[`${assetType}TableViewRowsCountByInProgress`].value}
      />
    )
    
    const Accepted = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='Accepted'
        filterBy={ENTITY_VERIFICATION_STATUS.accepted}
        isActive={ENTITY_VERIFICATION_STATUS.accepted === value}
        numberOfEntity={domain.ui[`${assetType}TableViewRowsCountByAccepted`].value}
      />
    )
    
    const Ignore = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='Ignored'
        filterBy={ENTITY_VERIFICATION_STATUS.ignore}
        isActive={ENTITY_VERIFICATION_STATUS.ignore === value}
        numberOfEntity={ignoredCount}
      />
    )
    
    const Fixed = (
      <CreateInfrastructurePageFilterToggleGroupItem 
        name='Fixed'
        filterBy={ENTITY_VERIFICATION_STATUS.fixed}
        isActive={ENTITY_VERIFICATION_STATUS.fixed === value}
        numberOfEntity={domain.ui[`${assetType}TableViewRowsCountByFixed`].value}
      />
    )

    return (
      <Grid container sx={{ height: '60px', width: '615px' }}>
          <ToggleButtonGroup
            value={value}
            exclusive
            onChange={(event) => domain.ui.filterByStatusHandler(assetType, event)}
            aria-label="text alignment"
          >
            {All}
            {New}
            {inProgress}
            {Fixed}
            {Accepted}
            {Ignore}
          </ToggleButtonGroup>
      </Grid>
    );
  },
);

interface IInfrastructurePageFilterToggleGroupItemProps {
  name: string,
  filterBy: InfrastructureAssetVerificationStatuses | 'all',
  isActive: boolean,
  numberOfEntity: number,
}

const CreateInfrastructurePageFilterToggleGroupItem = observer(({
  name,
  filterBy,
  isActive,
  numberOfEntity,
}: IInfrastructurePageFilterToggleGroupItemProps) => {
  const theme: Theme = useTheme();
  return (
    <ToggleButton      
      aria-selected={isActive}
      value={filterBy}
    >
      <Box
      >
        <Typography variant="body2" color={isActive ? 'text.primary' : theme.palette.text.secondary}>
          {name}
        </Typography>
      </Box>
      <EasmBadge numberOfEntity={numberOfEntity} />
    </ToggleButton>
  );
});

And this is the filterByStatusHandler function which I'm calling while toggling between tabs:

filterByStatusHandler = (
  assetType: keyof IInfrastructureTableViewFiltersByAssetCategory,
  event: React.MouseEvent<HTMLInputElement>,
): void => {
  const {
    currentTarget: { value: newValue },
  } = event;
  if (!newValue) return;
  const entity = this[`${assetType}TableView`]?.entity;
  
  //@ts-ignore
  this[`${assetType}TableView`]?.setEntity({ ...entity, filter: newValue });
  
  return;
};

Now, I have a problem — while I'm on the all tab (while domain.ui[`${assetType}TableView`]?.entity.filter === 'all'),
everything is working fine: each tab data is correctly filtered, counters are correctly counted and when I toggle between tabs (only if I start from 'all' tab), data and counters remain correct.
But if I apply any search or filter in the Data Grid component while on any other tab ('new', 'inProgress', 'fixed' or 'accepted'),
all tabs data arrays are incorrectly filtered and counters are incorrectly counted (including also breaking filters and counter for 'all' tab data).

I've tried to separately filter data and toggle view for different statuses, but, eventually, either the counters are wrong while updated simultaneously (on any tab other than on 'all'), or, the counters are fine, but only when I pass unfiltered data into the rows prop of the Data Grid (but in that case, the views are wrong, since they are not filtered by status).
I've tried this to fix the latter like this:

if (this[`${assetType}TableView`].entity.filter === 'all') {
  apiRef.current.updateRows(
    rowIds.map((rowId) => ({ id: rowId, ...filteredDataRowsByAll.find((item) => item.id === rowId) })),
  );
}

if (this[`${assetType}TableView`].entity.filter === ENTITY_VERIFICATION_STATUS.new) {
  apiRef.current.updateRows(
    rowIds.map((rowId) => ({ id: rowId, ...filteredDataRowsByNew.find((item) => item.id === rowId) })),
  );
}

if (this[`${assetType}TableView`].entity.filter === ENTITY_VERIFICATION_STATUS.inProgress) {
  apiRef.current.updateRows(
    rowIds.map((rowId) => ({ id: rowId, ...filteredDataRowsByInProgress.find((item) => item.id === rowId) })),
  );
}

if (this[`${assetType}TableView`].entity.filter === ENTITY_VERIFICATION_STATUS.fixed) {
  apiRef.current.updateRows(
    rowIds.map((rowId) => ({ id: rowId, ...filteredDataRowsByFixed.find((item) => item.id === rowId) })),
  );
}

if (this[`${assetType}TableView`].entity.filter === ENTITY_VERIFICATION_STATUS.accepted) {
  apiRef.current.updateRows(
    rowIds.map((rowId) => ({ id: rowId, ...filteredDataRowsByAccepted.find((item) => item.id === rowId) })),
  );
}

So, basically, I just tried to update the rows of the Data Grid for each of the tab manually without affecting the data passed into the rows prop of the Data Grid,
which, in turn, affects calculating the counters.

And for that, also, I've tried saving the state from the api.current.state while I have unfiltered data on the all tab, and count filters for other tabs based on that saved state, and not the apiRef changed in the future, but that also didn't work:

useEffect(() => {
  const filterValue = domain.ui.sslTableView.entity.filter;
  const newFilterModel: GridFilterModel = {
    items: filterValue === 'all' ? [] : [{ field: 'verificationStatus', operator: 'equals', value: filterValue }],
  };
  setFilterModel(newFilterModel);
}, [domain.ui.sslTableView.entity.filterActive]);

const debouncedHandleStateChange = useCallback(
  debounce(() => {
    if (domain.ui.sslTableView.entity.filter === 'all') {
      const currentState = apiRef.current.state;
      setSavedGridState(currentState);
      domain.ui.handleStateChange(rows, 'ssl', currentState);
    } else {
      if (savedGridState) {
        setTimeout(() => {
          domain.ui.handleStateChange(rows, 'ssl', savedGridState);
        }, 100);
      }
    }
  }, 500),
  [rows, savedGridState, domain.ui.sslTableView.entity.filterActive]
);

So, my guess is — either what I'm trying to achieve is not achievable based on how MUI X Data Grid Pro works right now,

or, if I'm wrong and/or maybe am close to a working solution — I would very much appreciate any help and advice.

Thanks!

p.s.

I'm already using MobX for states in my logic domain stores (all those this.ui.tableView.entity and this[`${assetType}TableView`]?.setEntity({ ...entity, filter: newValue }) lines with states and methods).

But I also have been thinking of trying to use React Context API for this case, just couldn't wrap my head around of how to do that properly in this case and how it would actually help with the Data Grid underlying issue, of when any change in the data passed into the rows prop of the Data Grid component is causing changes of filter counters on all tabs.

I also tried zustand, but it hasn't changed much with this issue in particular, unfortunately.

Your environment

`npx @mui/envinfo`

I used Brave browser (it's Chromium based)

System:
    OS: macOS 14.4.1
  Binaries:
    Node: 18.18.2 - ~/.nvm/versions/node/v18.18.2/bin/node
    npm: 9.8.1 - ~/src/frontend/node_modules/.bin/npm
    pnpm: 9.2.0 - ~/.nvm/versions/node/v16.18.0/bin/pnpm
  Browsers:
    Chrome: 125.0.6422.142
    Edge: Not Found
    Safari: 17.4.1
    Brave: 1.66.118 Chromium: 125.0.6422.147 (Official Build) (arm64) 
  npmPackages:
    @emotion/react: 11.11.3 => 11.11.3
    @emotion/styled: 11.11.0 => 11.11.0
    @mui/base: 5.0.0-beta.36 => 5.0.0-beta.36
    @mui/core-downloads-tracker:  5.15.10
    @mui/icons-material: 5.15.10 => 5.15.10
    @mui/lab: 5.0.0-alpha.132 => 5.0.0-alpha.132
    @mui/material: 5.15.10 => 5.15.10
    @mui/private-theming:  5.15.9
    @mui/styled-engine:  5.15.9
    @mui/styled-engine-sc: 6.0.0-alpha.16 => 6.0.0-alpha.16
    @mui/styles:  5.14.12
    @mui/system: 5.15.9 => 5.15.9
    @mui/types: 7.2.13 => 7.2.13
    @mui/utils:  5.15.9
    @mui/x-data-grid: 6.18.1 => 6.18.1
    @mui/x-data-grid-generator: 6.18.1 => 6.18.1
    @mui/x-data-grid-premium:  6.18.1
    @mui/x-data-grid-pro: 6.18.1 => 6.18.1
    @mui/x-date-pickers: 6.5.0 => 6.5.0
    @mui/x-license-pro:  6.10.2
    @types/react: 18.2.23 => 18.2.23
    react: 18.2.0 => 18.2.0
    react-dom: 18.2.0 => 18.2.0
    styled-components: 6.0.0-rc.1 => 6.0.0-rc.1
    typescript: 4.3.5 => 4.3.5

Search keywords: state management filtering conditional data for rows
Order ID: 89674

[michelengelen]

@davesagraf WOW 🤩
Well, if i understood your requirements correctly you are struggling with the filtering and counting, right?

First of all .. it looks like you are filtering manually, is that right?
If that's the case I would recommend to use the filtering mechanisms from the grid, since those are also a bit more performant.

I'll try out some things.
In the meantime: Could you maybe try to replicate that usecase in a live code example? That would be very helpful.

[davesagraf]

@michelengelen Hi! Thanks for the reply!

Okay, I'll try to make a live code example.

Looking forward to your suggestions.

Thank you!

[michelengelen]

Hey @davesagraf would this be something that potentially solves your usecase?

DEMO

[davesagraf]

Hi again @michelengelen !

Thank you so much for the demo! Yeah, it's very close to what I've been trying to achieve.

Just one more thing though, I need to handle and calculate data counters for different states of data based on the status filter as well (and those need to work simultaneously somehow)

I've taken your demo and tried to add some logic from my initial code here: new demo

But I get this error:
Maximum update depth exceeded. This can happen when a component repeatedly calls setState inside componentWillUpdate or componentDidUpdate. React limits the number of nested updates to prevent infinite loops.

[michelengelen]

Hey @davesagraf ... that demo is not public unfortunately

[davesagraf]

Hey again @michelengelen , thanks for pointing that out, the demo should be public now

[michelengelen]

I think knowing the count before applying the filter is currently not possible, but you could do it manually by counting a filtered array:

const array = [
  { status: 'OPEN' },
  { status: 'CLOSED' },
  { status: 'OPEN' },
  { status: 'CLOSED' },
  { status: 'OPEN' },
];

const count = array.filter(item => item.status === 'OPEN').length;

but for large data sets this would mean a heavy performance implication.

[michelengelen]

Is your usecase to show the expected amount on the button or do you want to show it when actively using the filter?

[davesagraf]

@michelengelen Thanks a lot for the tip!

Yes, basically, on the high level it would look something like this:
We have some initial unfiltered data.

There are 5 tabs, on each of which the initial data set is limited to only those rows where data item has the corresponding status.
On any of those tabs, we can use the Data Grid as we normally do, applying search and multifilters.

So, for any given tab, we can see, for example, how many of the items with the corresponding status have been left (rows visible) after applying a quickfilter or multifilters.

However, if we speak about counters and filter buttons showing them, I was trying to show the actual numbers of all the data filtered by status subsets with the filters currently applied.

So, for instance, we have the initial data of 10 items — 10 would be on the 'all' tab (filter button), 5 with a status of 'new', 3 'open', 1 'filled' and 1 'rejected'.

So those buttons would already show these initial numbers.

Then, if we apply a search or a multifilter value of say 'gmail' for the 'email' field, and 3 of 5 items with the 'new' status, 2 of 'open' and 1 'filled' have them, we should now have filter buttons with these values:
all (6), new (3), open (2) and filled (1) [we don't need to show 0 for the 'rejected'].

And my initial problem was that I was trying to achieve this behavior to be the same no matter on which tab we were when the filter was applied, but in my implementation that worked only on the first tab ('all' in our case), because it had the initial data.

[michelengelen]

Hey @davesagraf ... Just a follow up on this: I discussed this briefly with @cherniavskii and he came up with a quick PR (#13418) to resolve this issue. With this it should be possible to achieve what you are trying to implement easily!

I will see the PR through, so that you can start using it with the next release (by the end of the week).

[davesagraf]

@michelengelen @cherniavskii

Thank you so much, guys!

[davesagraf]

Hi @michelengelen !

I know you and @cherniavskii and other MUI X team members are working hard on a new feature for gitFilterState, so thanks a lot for your effort once again!

There's just something I wanted to show you while you're still working on that feature (maybe I'm too early for that, or just didn't get something, if so, sorry for that).

Here's a demo which is really close to what I've been aiming for from the very start, but it still has the same problem I've had which got me to open this issue.

So, when we're on the 'All' tab (the initial state), we see all the counts and if we apply any other filter in the multifilters, we see the correct updates for all the tabs/status buttons.

But, if we click on any of the rest of status buttons ('Open', 'Filled', 'Partially Filled' and 'Rejected'), all other tabs counts inherit the state of that active button we've just clicked.

So (again, sorry, if I'm too early for this) my question is — is there a way to make it something like this:
When we click any of the buttons among 'Filled', 'Open', 'Partially Filled' and 'Rejected', we don't update the counts based on the status filters for all the buttons, but only based on other filters (quickfilter or other multifilter values)

So, when we click on the filter buttons ('All', 'Filled', 'Open', 'Partially Filled' and 'Rejected') we only update the visible rows.

My goal is to update counts for all the tabs simultaneously, regardless of what 'tab' is active (which status button has been clicked) when we apply any other filter, meanwhile, correctly updating visual rows when we click any of those status buttons.

Thanks again!

[michelengelen]

@davesagraf this is exactly what you can do when the PR is merged! 👍🏼

[cherniavskii]

@davesagraf Ah, I see what you mean.
I didn't test it in combination with other filters, not sure why this happens. I'll look into it, thanks for the example!

[cherniavskii]

@davesagraf I've forked your demo with the latest Data Grid build from the PR: https://codesandbox.io/p/sandbox/pedantic-jasper-frf2zh

my question is — is there a way to make it something like this:
When we click any of the buttons among 'Filled', 'Open', 'Partially Filled' and 'Rejected', we don't update the counts based on the status filters for all the buttons, but only based on other filters (quickfilter or other multifilter values)

It is up to you how you combine the filters.
I've tweaked your demo, here's the result: https://codesandbox.io/p/sandbox/cool-hofstadter-48685v
Is this what you're looking for?

[davesagraf]

Hey @michelengelen !

That's great, thanks a lot!

[davesagraf]

Hey @cherniavskii !

Yes! That is exactly what I've been trying to achieve for the past week, but with the previous Data Grid logic "under the hood", that didn't seem possible.

Thank you so much!

Really looking forward to these updates!

[github-actions]

⚠️ This issue has been closed. If you have a similar problem but not exactly the same, please open a new issue.
Now, if you have additional information related to this issue or things that could help future readers, feel free to leave a comment.

@davesagraf: How did we do? Your experience with our support team matters to us. If you have a moment, please share your thoughts in this short Support Satisfaction survey.