virtual-scroll: integrate with relevant existing components

[mmalerba]

Integrate virtual-scrolling as an optional add-on for relevant existing components, including:

• Accordion
• Autocomplete
• Datepicker
• Grid List
• List
• Select
• Table
• Tree
• Drag & Drop

Part of each integration should include adding docs examples of how to set it up

[ahsan]

Hi, thanks for the awesome virtual scroll feature.
Any ETA for mat-tree?

[shlomiassaf]

I managed to implement virtual scroll with the table, working pretty amazing out of the box with both fixed and auto strategies.

I used the original CdkVirtualScrollViewport component and provided my "version" of CdkVirtualForOf (which should actually be the CdkTable + measuring capabilities)

I added 1 more strategy - NoStrategy and built my own directives put on the table to control which strategy to use.

I had some issues, most of them were easily solved, the most difficult thing to expect is the handling of meta rows, which are header and footer rows.

Handling of headers and footers require special treatment because they are not part of the datasource and thus mess up the entire flow, once you have multiple header/footer rows it get's messy quickly.

I solved all issues with size measurements when working with multi-header/footer table.

The only issues i'm facing now is sticky rows, which does not work because the virtual table now has a container that "offsets" the sticky row so position top 0 is no longer 0 after it was translated by the parent.

Working around this should also be easy if I had access to the function that sets the offset - I dont because it's private in CdkVirtualScrollViewport

[kiwikern]

I think integrating it with the Grid list would also be a great feature.

[IlCallo]

@shlomiassaf can you share the code you used or try a PR?
Virtual scroll is a thing in many are waiting to see integrated into MatTable component, a workaround while we wait for official support can be useful

[ahsan]

I needed Virtual Scroll in MatTree component, I hacked the MatList component to emulate behavior of MatTree as it already has the virtual scrolling capability. I did have to manage the indentations of individual nodes myself though.
Maybe a similar sort of approach can be used for MatTable component as well.

[shlomiassaf]

@IlCallo It would be difficult to share it right now, I need to clean some IP stuff from there.

The virtual scroll is also a part of a table component (on top of CdkTable) so it has some things that will not work as they are part of that table eco-system, things like global configuration, plugin integration etc...

I can confirm that it works, and works quite fast! for both Auto and Fixed size strategies.
I will try to extract it somehow, but I can't commit to a timeframe.

For now, I will try to describe how I did it:

This is my way of implementing it, there might be other ways!

First, the general layout we will use:

<cdk-virtual-scroll-viewport>
    <cdk-table></cdk-table>
</cdk-virtual-scroll-viewport>

The virtual scroll is external to the table.

Now we need to take care of 3 topics:

• Adjusting CdkVirtualScrollViewport to work with the table
• Providing our own version of CdkVirtualFor
• Adjusting strategies to work with the table

Adjusting CdkVirtualScrollViewport to work with the table

The general idea is to create a custom viewport component that inherits from CdkVirtualScrollViewport and apply minor adjustments so the table will work.

Our CdkVirtualScrollViewport will also control which strategy we use, replacing to our own table-targeted strategy when needed.

Providing our own version of CdkVirtualFor

We need to connect CdkVirtualScrollViewport with CdkTable, but CdkVirtualScrollViewport requires a CdkVirtualFor, which is a structural directive...

In general, CdkVirtualFor will render a subset (range) of rows from a DataSource and act upon changes in the datasource or range so it will always render the right subset.

CdkTable already does all of that, and some more...

We will use a simple class that mimics CdkVirtualFor while bridging the two components.

---

On a side-note, @mmalerba: CdkVirtualScrollViewport.attach(forOf: CdkVirtualForOf<any>): void; is probably narrow, forOf should proably be:

interface CdkVirtualScrollAdapter {
  dataStream: Observable<T[] | ReadonlyArray<T>>
  measureRangeSize(range: ListRange, orientation: 'horizontal' | 'vertical'): number;
}

---

Our CdkVirtualFor does 2 things:

Make sure sticky rows works

Because the table position is not static (transformed all over in the virtual viewport) the sticky rows
will not stick... we need to compensate for the transformation so their top root following the virtual top root offset. We do that by listening to offset changes (from CdkVirtualScrollViewport) and update the sticky rows with the offset.

Adjust the range to compensate for header/footer rows

Because the table might contain Header or Footer rows, we need to adjust the range accordingly. The virtual viewport is wrapping both header/footer rows and the actual table rows...

Our CdkVirtualFor adapter listens to renderedRangeStream changes from the viewport, and pass it on the the table (via CdkTable.viewChange.next(range)).

The viewport calculates the range using the strategy and it will return how many rows to render. This number is then used to extract rows from the data source. If we have a header row in view we need to reduce the range.

Adjusting strategies to work with the table

FixedSizeVirtualScrollStrategy works fine without changes, it's AutoSizeVirtualScrollStrategy that we need to amend.

The problem is not really in AutoSizeVirtualScrollStrategy but in CdkVirtualScrollViewport and how it emits the initial range, this is a known issue - See this comment by @mmalerba

To fix it I use a custom strategy that wraps AutoSizeVirtualScrollStrategy and ItemSizeAverager .
Basically, I use a TableItemSizeAverager that has access to the how many actual rows are rendered. When ItemSizeAverager.addSample() is called - if no rows are rendered it will use the default row height otherwise will work as is.

This could probably get solved differently, but because CdkVirtualScrollViewport has most of its logic methods private I had to go this way...

If this fix is not applied the average size will get very small values because it will get a large range of "rows" before rows are rendered... so the total height/rows will be small.

Hope it helps!!!

[shlomiassaf]

I managed to upload a small demo app I have for the table....

https://shlomiassaf.github.io/table-demo

Look at the "Demo" link on the left, it shows a large list with a virtual scroll (AUTO). You can set it to 100,000 rows about 23-24 columns.

The "All In One" link shows a virtual scroll with FIXED strategy.

It's a POC for all of you that want to use it.

Note that this is a quick and dirty demo app, expect bugs :)

[shlomiassaf]

OK, also managed to implement Drag and Drop using CdkDrag and my own version of CdkDropList.

See demo:

https://shlomiassaf.github.io/table-demo

It does both column and row d&d.

There is no "real" need to create a custom CdkDropList component, the material team just need to refactor it a bit so people can extend it (it's CDK after all)

Most of it is private and some functions are just big, if they port most to protected and split some functions (mainly _sortItem) I would be able to reuse it...

[CuriousDolphin]

it is possible to use virtual scroll with a grid list of element?.

[vrady]

OK, also managed to implement Drag and Drop using CdkDrag and my own version of CdkDropList.

See demo:

https://shlomiassaf.github.io/table-demo

It does both column and row d&d.

There is no "real" need to create a custom CdkDropList component, the material team just need to refactor it a bit so people can extend it (it's CDK after all)

Most of it is private and some functions are just big, if they port most to protected and split some functions (mainly _sortItem) I would be able to reuse it...

Hello @shlomiassaf , can you provide source code of your demo application? We need to figure out how to implement features that you show in demo. Thanks

[ghost]

I managed to upload a small demo app I have for the table....

https://shlomiassaf.github.io/table-demo

Look at the "Demo" link on the left, it shows a large list with a virtual scroll (AUTO). You can set it to 100,000 rows about 23-24 columns.

The "All In One" link shows a virtual scroll with FIXED strategy.

It's a POC for all of you that want to use it.

Note that this is a quick and dirty demo app, expect bugs :)

Hi,

It would be really great if you can share the code,

Thanks

[mmalerba]

@shlomiassaf Thanks for the detailed summary! That will be a great starting point for exploring integration with the table

[codestitch]

@shlomiassaf is there a way to check neg-table in your demo to experiment?

[nahgrin]

I got a basic version of virtual scroll working with the grid. I'll give a brief overview of what I did and try to come back and post a working example in a bit. I tried to follow what @shlomiassaf did and I ended up with a slightly different approach.

For the following code, I "borrowed" heavily from the material table examples. I'll try to describe what I did here and then just leave the code below to hopefully help answer any questions that my explanation leaves.

For the HTML, I wrapped the table element in the cdk-virtual-scroll-viewport as was suggested. However, I also had to modify the outlet for the row data so that it combined the cdkVirtualFor with the matRowDef. Instead of using the structural directive for the row, I expanded it out and kind of merged it with the cdkVirtualFor. Another important thing was that the datasource for the cdkVirtualFor is not the same one that is feeding the table. The rows observable is basically the true observable of the data in the grid while the dataSource observable is a filtered version of the rows for the table.

I created my own strategy for dealing with the virtual scroll in the table and it's mostly just an exceedingly simplified version of the FixedSizeVirtualScrollStrategy from @angular/cdk/scrolling. The reason I did this was that the FixedSizeVirtualScrollStrategy was producing some really weird rendering errors where the table would routinely display elements in the table off by a certain index. I think that it was causing the cdkVirtualFor and the mat-table to fight each other for rendering or something, but I'm not informed enough to say for sure. Other than that problem, the FixedSizeVirtualScrollStrategy can just be dropped in and will work without concern.

The component stitches the data in the table and the strategy together and creates the separate dataSource observable for the table. Every time that the index of the scroll is updated it modifies the slice of the array so that the table only renders the piece of the table that should be in view.

That's basically what I've done and it is working pretty well for me. If anyone has any insight into getting the FixedSizeVirtualScrollStrategy, that would be wonderful.

table.component.html

<cdk-virtual-scroll-viewport [style.height.px]="gridHeight">
  <table mat-table [dataSource]="dataSource" class="mat-elevation-z8">

    <ng-container matColumnDef="position">
      <th mat-header-cell *matHeaderCellDef> No. </th>
      <td mat-cell *matCellDef="let element"> {{element.position}} </td>
    </ng-container>

    <ng-container matColumnDef="name">
      <th mat-header-cell *matHeaderCellDef> Name </th>
      <td mat-cell *matCellDef="let element"> {{element.name}} </td>
    </ng-container>

    <ng-container matColumnDef="weight">
      <th mat-header-cell *matHeaderCellDef> Weight </th>
      <td mat-cell *matCellDef="let element"> {{element.weight}} </td>
    </ng-container>

    <ng-container matColumnDef="symbol">
      <th mat-header-cell *matHeaderCellDef> Symbol </th>
      <td mat-cell *matCellDef="let element"> {{element.symbol}} </td>
    </ng-container>

    <tr mat-header-row *matHeaderRowDef="displayedColumns"></tr>
    <ng-template let-row matRowDef cdkVirtualFor [matRowDefColumns]="displayedColumns"[cdkVirtualForOf]="rows">
      <tr mat-row></tr>
    </ng-template>
  </table>
</cdk-virtual-scroll-viewport>

table-vs-strategy.service.ts

@Injectable()
export class TableVirtualScrollStrategy implements VirtualScrollStrategy {

  private scrollHeight!: number;
  private scrollHeader!: number;
  private readonly indexChange = new Subject<number>();

  private viewport: CdkVirtualScrollViewport;

  public scrolledIndexChange: Observable<number>;

  constructor() {
    this.scrolledIndexChange = this.indexChange.asObservable().pipe(distinctUntilChanged());
  }

  public attach(viewport: CdkVirtualScrollViewport): void {
    this.viewport = viewport;
    this.onDataLengthChanged();
    this.updateContent(viewport);
  }

  public detach(): void {
    // no-op
  }

  public onContentScrolled(): void {
    this.updateContent(this.viewport);
  }

  public onDataLengthChanged(): void {
    this.viewport.setTotalContentSize(this.viewport.getDataLength() * this.scrollHeight);
  }

  public onContentRendered(): void {
    // no-op
  }

  public onRenderedOffsetChanged(): void {
    // no-op
  }

  public scrollToIndex(index: number, behavior: ScrollBehavior): void {
    // no-op
  }

  public setScrollHeight(rowHeight: number, headerHeight: number) {
    this.scrollHeight = rowHeight;
    this.scrollHeader = headerHeight;
    this.updateContent(this.viewport);
  }

  private updateContent(viewport: CdkVirtualScrollViewport) {
    const newIndex = Math.max(0, Math.round((viewport.measureScrollOffset() - this.scrollHeader) / this.scrollHeight) - 2);
    viewport.setRenderedContentOffset(this.scrollHeight * newIndex);
    this.indexChange.next(
      Math.round((viewport.measureScrollOffset() - this.scrollHeader) / this.scrollHeight) + 1
    );
  }
}

table.component.ts

@Component({
  providers: [{
    provide: VIRTUAL_SCROLL_STRATEGY,
    useClass: TableVirtualScrollStrategy
  }],
  ...
})
export class TableComponent implements OnInit {

  // Manually set the amount of buffer and the height of the table elements
  static BUFFER_SIZE = 3;
  rowHeight = 48;
  headerHeight = 56;
  
  rows: Observable<Array<any>> = of(new Array(1000).fill({position: 1, name: 'Hydrogen', weight: 1.0079, symbol: 'H'}));

  displayedColumns: string[] = ['position', 'name', 'weight', 'symbol'];

  dataSource: Array<any>;

  gridHeight = 400;

  constructor(@Inject(VIRTUAL_SCROLL_STRATEGY) private readonly scrollStrategy: TableVirtualScrollStrategy) {}

  public ngOnInit() {
    const range = Math.ceil(this.gridHeight / this.rowHeight) + TableComponent.BUFFER_SIZE;
    this.scrollStrategy.setScrollHeight(this.rowHeight, this.headerHeight);

    this.dataSource = combineLatest([this.rows, this.scrollStrategy.scrolledIndexChange]).pipe(
      map((value: any) => {

        // Determine the start and end rendered range
        const start = Math.max(0, value[1] - GridComponent.BUFFER_SIZE);
        const end = Math.min(value[0].length, value[1] + range);

        // Update the datasource for the rendered range of data
        return value[0].slice(start, end);
      })
    );
  }
}

[garrettld]

@nahgrin I hope you don't mind, I've thrown your code into StackBlitz so that I could play around with it.

https://stackblitz.com/edit/nahgrin-virtual-scroll-table

There was a tiny bit of cleanup, but it pretty much just worked. Thanks so much!

[nahgrin]

@garrettld Awesome! Thanks a bunch for doing that!

[shlomiassaf]

@nahgrin Nice work!

The problem is in the header/footer rows, you need to take them into account when you calculate the range from the data source.

For example, if I have 5 header rows and 1000 items.
Let's say I can fit 10 rows in my viewport and for simplicity, there is no buffer.

When i'm in 0 scroll offset I can't return 10 rows because 5 are used by headers... same goes for footer rows.

You need to calculate the header/footer rows and their visible height in the view, remove that and use the height left to calculate what is the actual range.

Here's an example with 5 header rows that breaks it:

https://stackblitz.com/edit/nahgrin-virtual-scroll-table-wdlqlm?file=src%2Fapp%2Ftable%2Ftable.component.ts

Of course there are more things to take care of, sticky rows and AutoSize strategy...

You might want to refactor TableVirtualScrollStrategy into a directive. You can
put that directive on the table and then inject the table to it. You will have access to the table instance and the CdkVirtualScrollViewPort instance.

On that directive you can define all the heights... and you can also use a special input for the datasource, the directive will take that datasource and assign it to the table so it's a real plugin.

[ghost]

@nahgrin Tremendous work. Thanks for your efforts.

[Jadamae77]

@garrettld great solution! I am attempting to implement it in my app, but it seems to be retaining views that have been rendered but are no longer in the viewport. Do you know what I would need to change to prevent this?

Edit Upon further investigation, it appears it might not be view caching. I have a input with an autocomplete in each row, but sometimes the autocomplete panel doesn't appear until you scroll the table, and sometimes they stack up?

[aalbericio]

I would love to hear what the Angular Material team has to say about the virtual scroll stuff since it's a 3.5 year issue that should have been addressed at day 1 of their development, not day 1 of this issue but day 1 of Angular Material :(

I still have faith on you, guys!

[Jadamae77]

@garrettld The last row doesn't seem to be displayed in your stackblitz. I'm guessing due to the area of the viewport taken up by the sticky header?

[weikhang95]

@Jadamae77

@garrettld The last row doesn't seem to be displayed in your stackblitz. I'm guessing due to the area of the viewport taken up by the sticky header?

Have you found the solution? I also having this issue.

[Jadamae77]

@weikhang95

I added + 1 to the getDataLength(): number {} fn in the table-virtual-scroll-viewport.component.ts.

   /** Gets the length of the data bound to this viewport (in number of items). */
   getDataLength(): number {
     return this._dataLength + 1;
   }

[apramendorfer]

So it is 2022 and the angular material table still does not support virtual scroll officially?

[zolakt]

It's incredible that this is opened for 4+ years, and it hasn't made any further than "under consideration"... What is there to consider? This is essential functionality for any app dealing with big data. Not every app is as simple as your "heroes example".

I guess it will be another 4 years just to put it on the roadmap.
Maybe in a few decades it actually gets fixed.

[richardsengers]

Any news about this would be very much appreciated

well....some "news" can be found here:
https://github.com/angular/components/projects/20#card-7669022

[JoranLive]

I hope this page implements virtual scroll as it could be still open in 20 years...

[Azbesciak]

Hello guys, anything new? this ticket has 5 years already.

[alixroyere]

Hello, note that with material 3 color enhancements, angular mat autocomplete now colours the selected value. Having a cdk virtual scroll viewport inside a mat autocomplete to handle mat options, is now colouring randomly some values. I don't know how the selected value is recognized, but this is weird

[awdorrin]

@alixroyere I just ran into this yesterday, the 'randomness' seems that if say, the 2nd item in your mat-option list is selected, that when the virtual scroll happens, the 2nd item in the next batch remains selected.
I don't know how to fix this so went back to slicing the mat-options to the first 10 or 20 items, rather than displaying all 3600 items (in my case)
Still, would really like to get this working, as our users don't like the truncated list.

[giovannidias5a]

5 years and nothing, wow!

[nonconceptual]

Based on the linked project, it's moving along, just taking time.

[lautarobock]

I've been using it for trees for a long time.

[yenmangu]

I cannot believe this ticket has been open for 5 years...

[labeled]

We're going for 6 years! :D

[awdorrin]

6 years and 12 'major' releases. Personally I think they should start slowing the version rolling and focusing on quality a bit more...

[lautarobock]

I think it is still alive as a joke. You can take a look at the documentation https://material.angular.io/cdk/scrolling/overview you will see that is is pretty flexible to adapt it to most of the use cases.

[DM-be]

Would the new defer also provide options to help with viewport optimisations? For example, only render a column in a table when its in viewport?

[zolakt]

6 years and 12 'major' releases. Personally I think they should start slowing the version rolling and focusing on quality a bit more...

They never cared about quality, why should they start now....
It's much easier to completely ignore the whole user base and do your own thing.
When you have 5+ years tickets, no one actually expects that you know how to fix them.
Ignorance over competence, the Angular team moto.

[georgiee]

I've watched this issue for many years as I wanted to use virtual-scroll with a list to create a custom dropdown list. In the end that dropdown worked fine for years without it. Sure, it hit some edge cases with too many items in the list, but we managed. Right now I don't use this feature for anything else.

Of course it would be absolutely great if someone would integrate virtual-scroll in the other cdk components. The fact that nothing happened in the past 6 years states two facts: Inside Google this is probably not asked for, so nobody contributes from that side. We, the community, need to accept that as a fact. Nobody ever promised us this, so why are some of us pretending as if they did?

What's left? The community should fill that gap and may contribute something. Instead, I see folks venting their frustration instead of helping moving forward with that topic. In this gigantic thread (150 comments today), there are only three persons, @shlomiassaf & @nahgrin & @piernik, who ever tried.

Everyone else? Asking, begging, or worse, complaining. You people are a burden to the open source community. You take and have zero respect for its maintainers and the work they put into it.

As said, I don't rely on virtual-scroll these days. That's why I'm not in the position to contribute code or any work any concept, but I'm willing to outline some ideas for the folks who depend on virtual-scroll.

1. Create a poll, to understand what are the most sought integrations. Because that list of 9 items, it's too big to act on.
2. Describe your current requirement and where exactly you want to integrate virtual-scroll. Maybe that''s a starting point to work on it as an actual code contribution.
3. Maybe it's your comfort zone, that make you complain rather then trying to actually extend ? Start digging into the CDK source code, it might spark something and grows you as a professional.
4. Do you feel forced by your work to use Angular & CDK. Do you feel pressure? Raise the topic that virtual-scroll gives you the issues for what is act. Maybe it's just fine without? If not, maybe you get time to work on an actual solution that you could contribute?
5. Break free from CDK/1. You might build a very long list with CDK that mimics a one-dimensional table? Check some table solutions like ag-grid, which come with their own virtualization solution.
6. Break free from CDK/2. CDK is great but if it comes into your way and it really, really hurts not to have virtual-scroll, think about creating a custom, very specialized, solution.
7. Accept and be pragmatic: Years ago I was annoyed when I learned that virtual-scroll was not supported in the other great cdk components (like the list), but I accepted it and it was good. Maybe it's a simple as letting go the "perfect performant solution".

Here the two contributions that made it into a dedicated package or repo for some updated visibility:

• https://www.npmjs.com/package/mat-virtual-table (@nahgrin)
• https://shlomiassaf.github.io/ngrid/ (@shlomiassaf)

Lastly, if I were maintaining this issue list I would immediately close this issue. I have respect for the original motivation to create the issue. Looking at it today, I see an issue being too wide, too unspecific. Over the years, it degenerated into a honeypot for unspecific comments. Whoever may try acting on a single topic (like the mentioned folks with their contributions) it gets buried quickly beneath complaints or begging hiding the so much needed exposure for those contributions.

Maybe @andrewseguin (who unassigned @mmalerba) could help here, close it with some guidance and recommendations? Or someone else from the maintaining team?

[awdorrin]

if I were maintaining this issue list I would immediately close this issue.

Closing unsolved issues serves no purpose other than to hide/ignore issues.

Someday, someone will have enough time, or get annoyed enough to pick up the issue and resolve it completely, or, people will pick up pieces one at a time and work them. But they won't find it if it is a closed issue.

Just because you don't have a need or desire to address the issue yourself, and now find the commentary annoying and don't want to read about it; how about you just roll your eyes and move on to another topic?

[albahrawy]

Hi guys,
I implemented a simple directive to introduce virtual scroll behavior to Mat-Table or cdk-table to use it in my work,
I published it to npm recently.
it supports sticky headers and footers.
just import it and add [virtual-scroll]-"datasource" rowHeight="your-custom-height" to mat-table,
<mat-table [virtual-scroll]-"datasource" rowHeight="40">
you can install it from here:
@ngx-nova/material-extensions-table-virtual-scroll
feel free to use it and give me a feed-back.

[DywiTom]

Google team, you can do it!