fix: allow page length 2500

[barredterra]

Multiple of our customers have recently expressed their struggle with loading many records: select page length 500, then klick load more, scroll down, load more, scroll down, ...

A long page length is useful for reports, when your filters result in a couple thousand records and you need to load and check (or print) all rows.

Why 2500? We always have 5x increments: 20, 100, 500, 2500

On my development site, the request take 115 ms for the server to process and about 4 seconds for the browser to render. This seems acceptable.

[ankush]

what do people do with 2500 records? 😳

Isn't filtering/searching far more useful for narrowing down? 2500 honestly is absurdly high page length 😅

[ankush]

A better option would be editable page length or 1000 maybe. 2500 is odd and a huge number 😅 People will load that many docs and wonder why everything is so slow. List view isn't quite designed to handle such high amount of elements.

[barredterra]

We got this request independently from two customers today. I guess clicking "show all" is just easier than coming up with adequate filtering or grouping strategies. Also, how to print a report with 5000 rows?

editable page length

people will just enter 9999

1000

Breaks the pattern of 5x and isn't much better than 500

List view isn't quite designed to handle such high amount of elements.

We can work on that :D

BTW, the original request was a "load all" button, which we objected. But I think 2500 is justifiable.

[ankush]

A better pattern for doing operations on everything matching filters is this:

Load All coupled with realtime refresh can be dangerous on DB with large tables 😅

[barredterra]

A better pattern for doing operations on everything matching filters is this

I don't see how this helps for use cases like:

• make report builder show all, say, 4200 rows with a correct totals row
• print a report (from report builder) with all, say, 4200 rows

Load All coupled with realtime refresh can be dangerous on DB with large tables

I pushed some fixes and UX improvements today that should avoid reloading all data or rebuilding the entire list HTML in some cases (just low hanging fruit for now. We can probably do more, if required):

• refactor: render list progressively #25066
• perf: don't process checks if there are none #25063
• perf: remove specific elements instead of re-rendering entire list #25078
• perf: "load more" and "increase page length" #25081
• perf: render list #25524

Edit: Ankush also did some performance improvements:

• perf: 🚀 rendering large data datatable#197
• fix: allow exporting large reports that can't be rendered #25395

[barredterra]

@ankush do you see any more bottlenecks we should tackle before this is good to go?

[ankush]

The only blocker with "apply action to all matching doc" is printing right? because prints are generated using client side HTML?

I really think we should attempt doing that server side 😅

Printing large reports also just doesn't work from UI. 😬 You'll hit max request size limit soon enough.

[ankush]

Merging for now.

[ankush]

Can you list all operations that aren't possible right now? Apart from printing.

[barredterra]

Features that currently require loading all rows:

• Showing correct values in the Total row of report builder

  Sums up the loaded data only. The difference is not obvious to users. Could also be solved by computing the total values in the backend.

• Using the columns' search fields in the report builder

  Searches through loaded data only. The difference is not obvious to users. Could also be solved by disabling this feature / allowing query filters only.

• Printing a report with all rows (via report builder)

  Could also be solved by printing in the backend. This would depend on the above point (disabling the columns' search fields, since they wouldn't get applied in the backend).

• Bulk-editing all rows

  Could better be solved via a "select all n rows" button