[DataGrid] New virtualization implementation

[m4theushw]

Breaking changes

• [DataGridPro] Remove the onViewportRowsChange prop and the viewportRowsChange event. Add a listener on the rowsScroll event as replacement. Check https://codesandbox.io/s/create-react-app-forked-ty1w7?file=/src/App.js.

  const apiRef = useGridApiRef();
  const prevRenderContext = React.useRef(null);
  
  React.useEffect(() => {
    return apiRef.current.subscribeEvent("rowsScroll", ({ renderContext }) => {
      if (
        !prevRenderContext.current ||
        renderContext.firstRowIdx !== prevRenderContext.current.firstRowIndex ||
        renderContext.lastRowIdx !== prevRenderContext.current.lastRowIndex
      ) {
        prevRenderContext.current = renderContext;
        const params = {
          firstRowIndex: renderContext.firstRowIndex,
          lastRowIndex: renderContext.lastRowIndex
        };
      }
    });
  }, [apiRef]);
  
  <DataGridPro apiRef={apiRef} />

• [DataGrid] The CSS classes .MuiDataGrid-window and .MuiDataGrid-windowContainer were removed.

• [DataGrid] The following CSS classes were renamed:

  • .MuiDataGrid-viewport was renamed to .MuiDataGrid-virtualScroller
  • .MuiDataGrid-dataContainer was renamed to .MuiDataGrid-virtualScrollerContent
  • .MuiDataGrid-renderingZone was renamed to .MuiDataGrid-virtualScrollerRenderZone

---

Preview: https://deploy-preview-2673--material-ui-x.netlify.app/components/data-grid/#commercial-version

EDIT: I renamed GridVirtualizedContainer to GridVirtualScroller to use a shorter name.

Diagnostic

Based mostly on #1933, the current virtualization has the following problems:

1. Complex to maintain, since it's made of two big hooks: useGridVirtualization and useGridContainerProps.
2. The scrollTop position is not respected, which leads to the famous "jump". [DataGrid] Virtualization not respecting scrollTop #1911
3. Scrolling is not as fluid as the native behavior. This happens because it watches the scroll position in one div and applies a CSS transformation to another div, which causes small lags or jittering.
4. Horizontal scroll performance is much worse than vertical scroll.
5. Blank space after the last row. [DataGrid] Blank space in the scroll container #414
6. The disableVirtualization prop is not a true non-virtualized mode as it still needs to apply a CSS transformation to sync the scroll position.

I didn't list all issues here but after the merge I'll go on each one closing them.

Proposed solution

Based on the problems mentioned above, this PR is proposing to add a new GridVirtualScroller which will control everything related to virtualization of rows and columns. The new virtualization doesn't reinvent the wheel and implements the same behavior of well known React libraries (e.g. react-virtualized). To begin to understand it, first we have to have in mind what a "render context" means here. The render context is simply a object containing the first and last rows/columns to render. Note that these values don't take into account overscan. The virtualization works by watching the scrollTop position of a scrollable container and each time a scroll event occurs, it calculates which should be the new first rendered row. Based on that value, the last rendered row is also calculated. After that, it compares how many rows (distance) the user has scrolled since the last scroll event. If this distance is equals or greater than rowThreshold, then it triggers a re-render and updates the render context. When actually rendering the rows, then overscan kicks in to render extra rows to avoid blank spaces. Since now the virtualization relays on the native scroll, there will be situations where no single re-render will occur, as the render context didn't change. The same behavior is also applied to the columns now, which solves the issue with horizontal scroll. After every scroll, the rowsScroll event is published with the scroll position and current render context.

For the virtualization of the column headers, it works almost the same way. The main difference is that instead of watching the scroll position of the main container, it listens to the rowsScroll event. Since the headers can't leverage the native scroll, it needs to do a CSS transformation on every scroll. See the trade-off below.

FPS comparison

Taken from running yarn benchmark:browser.

Before:

DataGridPro:
  Vertical scroll:
       Min: 24.186 fps (σ = 1.122)
       Max: 58.044 fps (σ = 0.506)
    Median: 52.182 fps (σ = 0.494)
      Mean: 46.125 fps (σ = 0.542)
  Horizontal scroll:
       Min: 11.632 fps (σ = 1.159)
       Max: 56.941 fps (σ = 1.053)
    Median: 21.28 fps (σ = 0.096)
      Mean: 22.142 fps (σ = 0.119)

After:

DataGridPro:
  Vertical scroll:
       Min: 25.424 fps (σ = 1.102)
       Max: 58.151 fps (σ = 0.413)
    Median: 45.909 fps (σ = 0.392)
      Mean: 45.733 fps (σ = 0.209)
  Horizontal scroll:
       Min: 17.585 fps (σ = 7.108)
       Max: 57.905 fps (σ = 0.295)
    Median: 46.875 fps (σ = 0.376)
      Mean: 44.008 fps (σ = 0.451)

Trade-off

The new virtualization is not perfect. The main issue which users might notice is the column headers appearing mis-aligned or out-of-sync with the their respective columns during scrolling. It can happen on a low-end device or when scrolling very fast in some browsers (I noticed it in Firefox). The rationale behind it is that the main container relays on the native scroll, which runs on a separate thread. In the other side, the virtualization of the column headers runs on the UI thread and this one is updated at intervals which might not match with the other thread. The ScrollSync component from react-virtualized suffers from the same problem. Here's an explanation from the author of this library: bvaughn/react-virtualized#369 (comment)

If that becomes an issue, we can add a prop to slow down the virtualization and force it to stay in sync with the headers. That's the last resort, only to be used if we can't argue why the native scroll is better.

Why it didn't happen with the old virtualization? Previously, the native scroll was not being used. On every scroll event, there was a CSS transformation to update the rendering zone position, so headers and columns stayed always in sync.

ZmsDbMOQBv.mp4

Components removed

• <GridStickyContainer /> - Hides anything that exceeds the grid's size. Despite the name, it renders the MuiDataGrid-viewport element. The same role is done now by the MuiDataGrid-virtualizedContainer element.
• <GridViewport /> - This is the old virtualization component. Replaced by the new GridVirtualScroller.
• <GridEmptyCell /> - When the width taken by the columns is less than the grid's width, this component fills the empty space of the row. The filler now is embedded into the row component.
• <GridRowCells /> - Returns the cells that make part of the row. Its functionality was merged into GridRow. [DataGrid] Remove GridRowCells #2811
• <GridDataContainer /> - Used to force the scrollbars to be visible. Has width and height equal to the content's size. It's represented now by the MuiDataGrid-content element rendered by the new virtualization component.
• <GridWindow /> - Used to display the content below the header, since MuiDataGrid-columnsContainer is absolute positioned. It was replaced by adding a top margin to the virtualization container.

Hooks removed

• useGridVirtualization - It watches the scroll position, calculates which rows and columns should be rendered and updates state.rendering.renderContext. It also moves the column headers when there's a horizontal scroll. It was replaced by GridVirtualizationContainer.
• useGridNoVirtualization - Like the previous one, but always rendering all rows and columns. This behavior is now embedded into GridVirtualizationContainer, which means that when virtualization is disabled there's no re-render in the content area.
• useRenderInfoLog - Watches the scroll position and logs everytime state.rendering.renderContext is changed. I could've migrated it to work with the new virtualization, but I found that this hook generates too much noise.

New props

• rowBuffer - On each scroll, it's calculated the first and last row that should be visible. This prop controls how many additional rows (a.k.a. "overscan") should be rendered before and after the calculated ones. Having these extra rows prevents the user from seeing a blank space when scrolling faster.
• rowThreshold - Controls how many rows from the buffer can be visible. It can also be understood as how late the virtualization will render new rows.
• columnThreshold - Same as rows but for columns. The columnBuffer prop already exists.

Future work

• To further improve virtualization, we could investigate in making the overscan "smarter". Currently, it's rendering the same amount of extra rows no matter the scroll direction. It could render them only in the same direction the user is scrolling.
• Decouple GridVirtualScroller from the state to allow reusability?
• The virtualization of the column headers is still spread accross several components. I plan to refactor this part to create a single component.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[m4theushw]

Keeping as internal state avoids having to re-render the entire grid.

[flaviendelangle]

Moving this one to be internal will help a lot if we want to migrate to a regular React state (useState) instead of a ref in the future 👍

[m4theushw]

This small change yielded one of the main performance achievements. Previously, on each render React had to create a new element since it didn't know one to reuse.

[m4theushw]

@DanailH What's the purpose of this prop? Maybe with the render context being passed in the rowsScroll params it might not be necessary anymore. I didn't have the time to look into how this prop works, so for now it's not being called.

[DanailH]

Ah yes - this is the first step of the server-side infinite loader. When this was created the virtualization worked in 'pages' so this called every time a new virtual page becomes visible and passes in the first and last row indexes.
I guess since we no longer have these "pages" this will be called each time the grid scrolls with the new first and last row indexes, or it can be when the overscan changes?

Update: Now that I got familiar with the PR I guess this prop rowBuffer is the one that is needed, so when this changes the mentioned callback should be called. That way it will provide the ability for developers to preload rows.

[m4theushw]

When you say 'virtual page' you can now replace it with 'render context'. They work almost the same way. My point is that I think this prop is not relevant anymore. If we need to know when the first and last row indexes have changed, it can be done only by comparing the reference to the render context. I'm already doing that for the columns headers in https://github.com/mui-org/material-ui-x/pull/2673/files#diff-8160a4255306366ba065fa7036042dba41c51e2602d8967706b95fe37aa655eaR116

[m4theushw]

Accessing Element.clientWidth or Element.scrollTop all the time is harmful to the performance since it may triggers layout recalculation. See: https://gist.github.com/paulirish/5d52fb081b3570c81e3a

[m4theushw]

Should we list the removed classes as breaking changes? I don't feel that the new ones are stable enough to be used as replacements.

[m4theushw]

Asserting virtualization stuff in unit-tests is hard. Each browser may use different translate values which would break our tests. I didn't put too much effort in testing here. I relayed more on argos-ci. I'll do another PR to clean all these tests and exercise every single bit of the virtualization.

[m4theushw]

A small regression I found. Dragging one column to the very edge, would scroll only to the left.

[m4theushw]

I don't think one component should share styles from another one.

[m4theushw]

Not used, but since there's a slot for the row, somebody might want to do something with it.

[github-actions]

This pull request has conflicts, please resolve those before we can evaluate the pull request.

[DanailH]

@m4theushw Maybe I missed something but why did you remove the onViewportRowsChange. I was left with the impression that we'll make it work and potentially change the name?