New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtualization fails above 50k recods in IE #22

Closed
milang opened this Issue Jan 30, 2010 · 34 comments

Comments

Projects
None yet
9 participants
@milang
Copy link
Contributor

milang commented Jan 30, 2010

To reproduce, update "example1-simple.html" data population loop to go to 100,000 (instead of 500). Now load the HTML file in IE (I tested in v8 and simulated v7). When you use the scrollbar to move to the very end, instead of "Task 99999" you see "Task 47721" in IE8 and "Task 53686" in simulated IE7.

I think this effect is caused by IE handling height of the grid-canvas element incorrectly. If you look at the above grid in Developer Tools, it says "height: 1342177.27px" which is clearly incorrect. This seems to be a limitation of IE and the only way to "fix" it (I think) is to also virtualize the grid-canvas height...

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Jan 30, 2010

If you think about it, a "virtualization" of the scrollbar is already taking place in the browser (because the scroll thumb won't go below certain size for usability reasons, there comes a time when the ratio thubSize:scrollSize is not equal to the ratio viewportHeight:containerHeight -- for very large container heights the thumb size would have to be sub-pixel to truly represent how high the viewport is relative to the container). I "feel" we should be able to apply the same mechanism as well. SlickGrid's advantage is that the rows are positioned using "top:value" style. So the problem really is how to compute the value.

Right now, the computation is pretty straightforward. You know the height of rows, you know the scroll position, so it's simple to compute what rows are in viewport and what is their offset in the scroll container.

Let's consider now the row is 30px high and we have 100k records. That means computed height of the scroll container is 3 million pixels. We agree that this is too much for IE. However, what if we set the scroll container height to only 1 million pixels and applied coeficient of 3 (3mil/1mil) to all computations? I.e. if the reported scroll position is 500k, for us it would be 500k * 3 (to determine what rows to display).

I think there are two important scenarios: 1. scroll position changes, grid needs to display items at the approximate location the scroll thumb represents (as I said above, for large heights the thumb already only approximates position); 2. grid changes displayed records (e.g. cursor moves out of viewport, or clients calls function to bring a record to view), scroll thumb needs to be moved to a position that approximates position of the displayed items.

I (again) feel the 2nd scenario is more difficult and would probably require "scroll position offset". But I really need to build a working prototype because I'm just guessing at this point :-).

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jan 30, 2010

The problem isn't the logic behind the virtualization and not even the implementation.
Let me clarify what I meant there.
There are 2 reasons why other similar virtual rendering grids that have a separate scrollbar control driving the canvas aren't nearly as fast and as smooth as SlickGrid. The first is that continuously updating absolute positions of multiple elements is slow. The second is that browsers throttle firing the onscroll event for the scrolled content used to create the scrollbar we want. The result is a lag that you can see in the SlickGrid when you do synchronized horizontal scrolling of headers and cells. Some browsers, like Firefox, are particularly bad at that.

SlickGrid uses the fact that the browser can do bitmap caching on scrollable panes and can respond to user moving the scrollbar very quickly with minimal lag. All SlickGrid has to do is to step in at the right times and clean up old rows and put the new ones in the right place.

Virtualizing the scrollbar will negate that advantage. The workaround I was talking about involved breaking the canvas into 1M-pixel "pages" and virtualizing the transition between them while still using native scrolling while withing a page.

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Jan 30, 2010

I see your point. The grid is nice and fast because you're allowing the browser to just "uncover" rows during scrolling, your main task being to make sure it has something to uncover (as opposed to moving things around using absolute positioning).

To preserve the speed, I don't really see a straightforward solution. Do you think the hybrid solution combining virtual pages and native scrolling within a page would work?

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jan 31, 2010

It's doable, but may get messy. Perhaps there are alternative routes there.
The limit seems to be in the CSS implementation. While you cannot set numeric style values to anything greater than 0x123456, the rest of the rendering engine doesn't have any issues with greater values (at least up to ~10M pixels).

For example, you can create 10 individual 1M-pixel-high elements and that will force the container to be 10M pixels high:

    var o = document.getElementById("mydiv");

    for (var i=0; i<10; i++) {
        var x = document.createElement("div");
        x.style.background = "yellow";
        x.style.height = "1000000px";
        o.appendChild(x);
    }

    // reports 10M pixels correctly
    alert(o.clientHeight);

(changing the loop limit to >10 breaks everything)

We can use this to force the height of the scrollable pane to correct values, but we still have the problem of not being able to set the "top" of the individual row to put it in the right place. Need to experiment with it some more.

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Feb 6, 2010

Just for reference, another type of failure due to height overflow is documented in issue #23 (this time in Firefox 3.6+).

@ghost

This comment has been minimized.

Copy link

ghost commented Feb 6, 2010

Have we checked that the Firefox 3.6+ issue can be resolved in the same way, by forcing container height with multiple children instead of CSS?

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Feb 8, 2010

At this point, this is just brainstorming. How important is this anyway? I mean, I realize that it is, but what is the priority?

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Feb 8, 2010

Bragging rights important? :-)))

"We are the only grid on The Net that can handle arbitrary row count" ;-)

There's always the pager you can fall back on. I would be a bit more comfortable if the cutoff point was in hundreds-of-thousands rows range (for 30px row height), but the 40k or so is ok. For me the priority of this issue is medium-low.

@ghost

This comment has been minimized.

Copy link

ghost commented Feb 8, 2010

The priority for us is very high because we chose to use slickgrid precisely for its ability to handle "arbitrary row count". For us, "arbitrary row count" might go up to hundreds of millions of rows.

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Feb 8, 2010

Can't quite imagine the scrollbar being useful at those row counts.
But I hear you guys. Will do some experimenting in the coming weeks.

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Feb 8, 2010

I don't think the current implementation will ever be able to handle hundreds of millions of rows. Sooner or later, height limit will rear its ugly head in every browser.

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Feb 15, 2010

Just pushed an experimental fix using the hack discussed above - http://github.com/mleibman/SlickGrid/commit/1c4860abcb9247011bb386b1e665f66e862b2aa8

There is still an upper limit there that varies with browsers. Chrome fares the best. IE8 breaks somewhere around 200K rows. FF is the weird one here. In the formatting example, it handles 1M rows just fine, but breaks on 400K. I suspect there is an internal conversion to floating point in position calculations there.

All-in-all, this does raise the upper limit considerably, but it still breaks down pretty quickly.
David, I'm interested in hearing your thoughts on how you plan to present the 100M-row grid to the user and still maintain any sort of reasonable navigation. The way I see it, you'd be using the scrollbar mostly for rough/relative navigation since the precision there would be extremely low (1px = hundreds of thousands of rows). The only remaining useful functionality there would be page up/down and row up/down using the scrollbar arrows. With that in mind, a combination of a graphical pager component (ex. http://www.extjs.com/deploy/dev/examples/grid/progress-bar-pager.html) and a native scrollbar for in-page navigation would work much better. You can still have a relatively large page size - maybe around 10K or 100K rows.

Thoughts?

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Feb 15, 2010

Another good example of a slider pager - http://www.extjs.com/deploy/dev/examples/grid/sliding-pager.html

@dlee

This comment has been minimized.

Copy link

dlee commented Mar 30, 2010

Is the 200K/400K limit even after the workaround? That's no good... I wonder if a third layer (divs of divs of divs) might work :)

You've understood my use-case perfectly: the grid is so that users can just navigate the data roughly. I'd guess the most useful regions are the beginning and the end, but we still want to provide a grid that supports the whole data.

As for pagers and sliders, the reason we chose to use slickgrid in the first place was to avoid the use of pagers and sliders.

@milang

This comment has been minimized.

Copy link
Contributor Author

milang commented Mar 31, 2010

Other than virtualizing scroll bar, it isn't possible to guarantee unlimited range of a virtual grid. Sooner or later browser height limitations will be encountered. And in case of SlickGrid virtualization of scroll bar is not easy/possible as it is using scrollbar offset to position rows before/after current viewport so that they are just "uncovered" by browser as you scroll down.

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jun 26, 2010

I think I have a solution - http://jsfiddle.net/SDa2B/
It's going to take a while to integrate this into SlickGrid, and there are areas that need to be figured out still, but I believe this does solve the problem.

Comments?

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jun 26, 2010

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jun 28, 2010

Implemented in a separate branch for now.
http://github.com/mleibman/SlickGrid/tree/unlimited-rows

Please test.

@mleibman

This comment has been minimized.

Copy link
Owner

mleibman commented Jul 2, 2010

Branch merged in. Closing :)

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Feb 24, 2016

Wanted to say thank you for this! Even six years later, this proved useful. I ported this chunk of logic over to React and it's working very nicely. https://github.com/Extensis/slick-scroll

@jouni

This comment has been minimized.

Copy link

jouni commented Mar 5, 2016

Just looked at the fiddle also, and works brilliantly, even after six years 😄

@quicksnap you can overcome the Safari momentum scrolling issue by adding pointer-events: none; to the content/rows during scrolling.

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Mar 6, 2016

@JouniK Holy crap, I'm going to try that out! Thanks!

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Mar 6, 2016

It works! Thank you so much @JouniK!

@quicksnap

This comment has been minimized.

Copy link

quicksnap commented Mar 6, 2016

@incidiuz

This comment has been minimized.

Copy link

incidiuz commented Jan 27, 2017

hi, can i build sclickgrid programmatically?

@6pac

This comment has been minimized.

Copy link

6pac commented Jan 27, 2017

There is an npm package: npm install slickgrid

But see the manifesto

@incidiuz

This comment has been minimized.

Copy link

incidiuz commented Jan 27, 2017

i want to create slickgrid with my data but the scroll is horizontal

@incidiuz

This comment has been minimized.

Copy link

incidiuz commented Jan 27, 2017

or this http://jsfiddle.net/SDa2B/263/ any example of horizontal scroll?

@6pac

This comment has been minimized.

Copy link

6pac commented Jan 27, 2017

don't think that's possible. SlickGrid uses the browser built in scroll bar.

If you still want to use SlickGrid my repo is current, check out the readme. this repo is no longer active. download the zip package. you should be able to run the examples (in the examples folder) locally. if you still have problems, please open an issue there.

@incidiuz

This comment has been minimized.

Copy link

incidiuz commented Jan 27, 2017

i will open new issue for example scroll. what i can do that?

@incidiuz

This comment has been minimized.

Copy link

incidiuz commented Jan 27, 2017

scroll horizontal i mean

@6pac

This comment has been minimized.

Copy link

6pac commented Jan 27, 2017

open an issue in my repo.
but i don't know what you mean by horizontal scroll, you'll need to explain that. If you mean using a horizontal scroll bar to scroll through data, you definitely can't do that with any version of SlickGrid.

@thadguidry

This comment has been minimized.

Copy link

thadguidry commented Oct 28, 2017

@6pac Horizontal scrolling is also fast and done very well in SlickGrid http://mleibman.github.io/SlickGrid/examples/example-spreadsheet.html

@6pac

This comment has been minimized.

Copy link

6pac commented Oct 29, 2017

@thadguidry sure, that's scrolling through column data horizontally, using the browser's built in scrolling. what i meant is, you can't scroll vertically through the row data, using a horizontal scroll bar. that's how i interpreted this request. i might be wrong :-)

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment