[RFC][Tree data] Lazy-load row children properties

[MBilalShafi]

This discussion discusses briefly the trade-offs and approaches under consideration for the implementation of properties for lazy load row children i.e #3377

Approach:

The idea is to make it work in the hybrid mode, i.e. users can provide initial treeData on the client side and they can provide two extra props using which they can lazy-load row children for certain rows, but they can still use methods like setRows, updateRows and even update rows prop on the client side, here are the two extra props as suggested in the original ticket:

hasChildrenOnServer?: (row: GridRowModel) => boolean;
fetchRowChildren?: (row: GridRowModel) => Promise<GridRowModel[]>;

Now, for parent rows (mostly those with rowNode.type === 'group') there could be these possibilities:

1. The row is client side and all the children are provided in the beginning (hasChildrenOnServer returns false for this row), this one will keep working as current.
2. The row is server-side, no children are provided in the beginning but hasChildrenOnServer returns true and we could do one of the following:
   • Add a dummy placeholder child row during rowTreeCreation to this row for it to be declared as rowNode.type === group, on fetching the data we can _delete this row and replace with the fetched rows.
   • We add a parameter in rowNode (like a boolean rowNode.serverSideData) which will be used to show expand icon in the row although its current rowNode.type is leaf, on fetching the new rows the rowNode.type of this row will be recomputed to be group as it has now some children represented by path provided by getTreeDataPath
3. The row is hybrid, i.e some children are loaded in the beginning via rows prop but more could be loaded lazily as hasChildrenOnServer returns true for this row.
   • Should we even support this use case?
   • If yes, then what should be the trigger of fetching the server-side rows (row expansion? what if rows are automatically expanded by defaultGroupingExpansionDepth or isGroupExpandedByDefault, same applies to server-side only rows)

As this is not the final solution, a full fledge solution would require to have a data fetching layer like AGGrid has (context) and then we can define it something like treeDataMode="server" to fully make it server side.

Impact on Data Processors like Filtering, Sorting, etc

As this one is a quick solution that unlocks people to achieve something closer to the full fledge solution, I wanted to have all the other features like filtering, sorting, aggregation, etc work in the same way as on the client side, it's the same as if a user is doing apiRef.current.updateRows on the client side. It's more relatable as underlying, this approach will use the same partial row update API (apiRef.current.updateRows)
For example, on this demo filtering on client-side persists on adding new rows using (apiRef.current.updateRows)

Users can move filtering, sorting, etc to the server-side using filteringMode='server' for example, but it may not work ideally as we are only updating part of rows using apiRef.current.updateRows instead of updating all the rows for performance reasons.

Pros:

• Quick solution
• Unblocks a lot of users looking for this solution
• Will possibly provide feedback and clear road-blocks for a more sophisticated solution that involves a data layer
• Will allow to forsee any breaking changes for the complete solution so that we can make them in time.

Cons:

• Won't be sufficient for all the use-cases
• Will probably require some rework later on

Conclusion

The purpose of RFC is to bring everyone on the same page before the actual implementation. Any comments or feedback would be appreciated.

[cherniavskii]

As this one is a quick solution that unlocks people to achieve something closer to the full fledge solution, I wanted to have all the other features like filtering, sorting, aggregation, etc work in the same way as on the client side, it's the same as if a user is doing apiRef.current.updateRows on the client side. It's more relatable as underlying, this approach will use the same partial row update API (apiRef.current.updateRows)

This is a good start! But what if instead of building this into the data grid, we fix the tree data lazy loading example and extract the custom code into a hook?
Then it could be used like this:

import { DataGridPro, useGridApiRef, useTreeDataLazyLoading } from '@mui/x-data-grid-pro';

function Demo() {
  const apiRef = useGridApiRef();
  const [rows] = React.useState([]);

  const { groupingColDef } = useTreeDataLazyLoading({
    apiRef,
    getRows: async (treeDataPath) => {
      const data = await fetchData(treeDataPath);
      return data;
    },
  });

  return (
    <DataGrid
      treeData
      apiRef={apiRef}
      rows={rows}
      columns={columns}
      getTreeDataPath={getTreeDataPath}
      groupingColDef={groupingColDef}
    />
  )
}

I created a sandbox where I tried to implement it: https://codesandbox.io/s/beautiful-goldberg-6rm4rm?file=/demo.tsx

This could be a temporary solution without virtually no internal changes to the data grid. The client-side features will work with it.
In the meantime, we can start working on the proper server-side data source.

What do you think?

[MBilalShafi]

Thanks @cherniavskii for the quick response, and for coming up with the codesandbox POC, it is a great idea 💡 and could be a reasonable middle-ground before an actual bigger solution.

Although, from a user's perspective, both fetchRowChildren/hasChildrenOnServer and passing getRows to the useTreeDataLazyLoading hook look a bit similar, the latter definitely has an advantage in that we don't have to touch much of the internals of the grid. 🎉

But, I think it will have the same limitations as in the demo.
I might be mistaken here, but I assume that is the reason @flaviendelangle had a dedicated issue for adding properties that support lazy-loading, basically to provide an intermediate solution, built into the Grid which is more stable, supports more use-cases and has a better runtime performance. (Otherwise, an issue for only moving the functionality of a demo to the grid internal logic that also has fair community traction doesn't make a lot of sense, no? 😄 )

Here are some of the many limitations of the current demo that I could think of:

1. Does not react to defaultGroupingExpansionDepth or isGroupExpandedByDefault: Although this is a worth discussing topic of what's the best way to handle these props with server-side rows, I feel we can work some way if we go with the proposed solution.

2. When row (or descendant) count is not known: The APIs with dynamic data (constantly updating) can not predict rowCount/descendant count in the beginning, it's only at the fetch time we know the size of lazily loaded rows, I can not see flexibility to handle this use-case in the current demo.

3. Paginated lazy loading: (Related to point no. 2) Some dynamic APIs may update their data occasionally (for a specific parent) which needs to be refetched or due to large data size they want to even load children data in batches (I know it's a black box for now, but we can think about the possibilities around how to support close to this use-case)

4. Handle API errors: Network promises are likely to be rejected, one necessity is to allow the users to handle failed scenarios by using something like onChildrenLazyLoadError apart from fetchRowChildren/getRows or another workaround to let users handle the errors efficiently.

5. Not much customizable: Users can not easily add a loading icon in the row as a feedback that data is loading or customize its positioning etc

6. Not that performance optimized: As stated by demo itself, it has terrible performances on a real dataset, which we can invest a bit on how to optimize.

7. Hybrid mode: Current approach in demo assumes all the group rows will be updated on the server-side only, I can think of a couple of use-cases where both server and client side could populate certain tree branches. I am not exactly sure though how much community traction this one has, as I haven't seen this in any other Grid solutions (like AgGrid etc)

I will try to do a POC to check which of these points are achievable with the custom hook approach (but I'm not sure if adding a lot of logic to the hook will make it a mini project in itself 🤔).

I am not sure how much complexity a proper data source solution brings in and how much time it could take, so probably we can think of some intermediate solution till then that works for certain common use cases.

What do you think?

(I am trying to create a reasonable code example which we can relate to and discuss around)

[cherniavskii]

1. Does not react to defaultGroupingExpansionDepth or isGroupExpandedByDefault: Although this is a worth discussing topic of what's the best way to handle these props with server-side rows, I feel we can work some way if we go with the proposed solution.

Correct, it's not supported.

2. When row (or descendant) count is not known: The APIs with dynamic data (constantly updating) can not predict rowCount/descendant count in the beginning, it's only at the fetch time we know the size of lazily loaded rows, I can not see flexibility to handle this use-case in the current demo.

Do we need the row count? In the linked demo, we fetch the rows on a per-level basis. The same applies to the hasChildrenOnServer +fetchRowChildren approach. Infinite/lazy loading on each level can be supported with a proper server-side data source.

3. Paginated lazy loading: (Related to point no. 2) Some dynamic APIs may update their data occasionally (for a specific parent) which needs to be refetched or due to large data size they want to even load children data in batches (I know it's a black box for now, but we can think about the possibilities around how to support close to this use-case)

This will only be possible in a proper server-side data source implementation. We should definitely consider this use case when implementing it.

4. Handle API errors: Network promises are likely to be rejected, one necessity is to allow the users to handle failed scenarios by using something like onChildrenLazyLoadError apart from fetchRowChildren/getRows or another workaround to let users handle the errors efficiently.

This can be handled in the useTreeDataLazyLoading, basically if getRows throws - we don't mark the group as fetched, and will allow to expand it again and refetch - something like this https://codesandbox.io/s/brave-tristan-teqzty?file=/demo.tsx

5. Not much customizable: Users can not easily add a loading icon in the row as a feedback that data is loading or customize its positioning etc

Agree.

6. Not that performance optimized: As stated by demo itself, it has terrible performances on a real dataset, which we can invest a bit on how to optimize.

I think it's no longer the case, after row tree structure change

7. Hybrid mode: Current approach in demo assumes all the group rows will be updated on the server-side only, I can think of a couple of use-cases where both server and client side could populate certain tree branches. I am not exactly sure though how much community traction this one has, as I haven't seen this in any other Grid solutions (like AgGrid etc)

I think it might be too early to think about potential "hybrid mode" since we don't even have proper server mode support yet 😅

I am not sure how much complexity a proper data source solution brings in and how much time it could take, so probably we can think of some intermediate solution till then that works for certain common use cases.
What do you think?

I think the server-side data source is overdue, and it's also the reason we are looking for an intermediate solution.
The sooner we start working on the proper server-side data source, the better.
I'm reluctant to invest a lot of resources into the temporary solution, especially if it only supports treeData.

[MBilalShafi]

Do we need the row count?

Yes, to show the expand icon, we use descendantCount > 0 as a condition, (the reason why Sarah is not expandable and Thomas is)

The sooner we start working on the proper server-side data source, the better.

That's correct, the plan I had was to make this intermediate solution that is not a throwaway, but actually extend it when adding a data source that handles stuff related to BE API, basically to go with an iterative approach, as there could be breaking changes, in which case, users might have to wait until v7.

There could still be some rework though. So, we can surely go with the approach you suggested. I'll refine the demo, open a PR and start working on the fully server-side solution, I hope we don't need any breaking changes. 🤞

---

On second thought, I guess there have been no noticeable complaints about the complexity of this demo or even about it being broken, which means that not many people are using it, so instead of wrapping it in the hook, we can target to merge this PR which fixes the demo and start working on the proper solution for the tree-data lazy loading.

[cherniavskii]

Yes, to show the expand icon, we use descendantCount > 0 as a condition, (the reason why Sarah is not expandable and Thomas is)

Oh, right! We don't have to make this required, a simple boolean indicating whether the row has children would be enough:
https://codesandbox.io/s/eager-rain-g7n8e6?file=/demo.tsx

I'll refine the demo, open a PR and start working on the fully server-side solution, I hope we don't need any breaking changes. 🤞

Sounds great! I know it's a broad topic and it's unclear how to implement it, so feel free to ping me if you want to discuss/brainstorm this idea 🙂

instead of wrapping it in the hook, we can target to merge #8840 which fixes the demo and start working on the proper solution for the tree-data lazy loading.

Nice, I didn't realize there's a PR for it already 👍

[MBilalShafi]

it's a broad topic and it's unclear how to implement it, so feel free to ping me if you want to discuss/brainstorm this idea

Exactly, that's the reason I've opened up an exploration PR, trying to figure out if the proposed interface is extendible enough to tick all the dos. Feel free to hop in if you want 🙂