add support for arbitrarily-sized topics

[indirectlylit]

Observed behavior

Especially when generated from sushi chefs, it's feasible that topics can have an arbitrary number of resources in them. Kolibri currently assumes that the number is something that can be returned in a single API call and displayed all at once on a single page.

Browsing topics with thousands of items in Kolibri brings the app to a halt

Expected behavior

We should have a design and implementation that supports arbitrarily large topics.

User-facing consequences

Unexpected freezes

Steps to reproduce

@ivanistheone might be able to help supply a channel that will reproduce this

Context

0.10.0

---

cc @jtamiace @khangmach we might want to start thinking about the UX of this

[ivanistheone]

I don't have anything with lots of nodes per folder (since we explicity try to avoid this), but if you want a channel for stress testing, nothing beats the PBS Media channel: https://studio.learningequality.org/channels/bc016b653d145d479ff3fe31b9ebd05d/edit

Warning don't import whole thing, it's 170G:

[khangmach]

Have we considered the convention of loading more items the further the user scrolls downward? There have been talks of pagination to accommodate this large resource size issue. Was there leaning towards this as opposed to "loading more" strat because the system still has a high chance of freezing when the page loads more than the current max? (~25?)

[ivanistheone]

@khangmach I don't think Kolibri frontend will have a problem handling longer lists (at least not in the 100-500 items range). I believe the bottlenecks is more the backend issue preparing and shipping all the metadata in one shot.

The frontend could be a load-more-when-near-bottom interface, and the "pagination" could be just a backend implementation detail needed to get results piecemeal (e.g. in chunks of 50).

[jtamiace]

Yes, a couple dev meetings ago, there was talk about using the "infinite scroll" pattern.

If that still sounds feasible, then on the design end, we'd just need sufficient loading state messages and UI transitions as more items are displayed.

If these topics are going to be getting that huge also, I'm wondering if a "back to top" button and/or something like in-page text filters would be favorable add-ons, especially for mobile users. It'd be really frustrating to have to scroll down to topic item 123 on mobile if you wanted to go back and continue learning about something

[indirectlylit]

My understanding is that very large topics are discouraged during channel creation, so this will not be a common scenario to users.

If that's accurate, simply providing a paginated API on the backend and dynamic loading of new items infinite-scroll-style on the front-end should be sufficient.

However if we anticipate very large topics to be a common scenario, then we should explore ways of improving the UX for that case (back-to-top, filter, etc)

[jtamiace]

I'm worried that a sub-optimal finding experience due to large topics will discourage use entirely of a content source, but maybe we just need to get feedback first and see for ourselves what the experience would be like.

If filtering enhancements do need to be made, other areas to consider are the import/export and lessons/exams UI's

[khangmach]

Wrote some documentation & designed some early specs on this issue!

[indirectlylit]

just to clarify: my understanding from @khangmach is that the empty blocks are not loading states. We should imagine that they have content in them. (I was confused by this originally)

[rtibbles]

This should be feasible, we already do pagination for search queries, and this would be similar.

Also worth noting that after recent changes to content loading, the strain of loading huge numbers of content nodes is decreased, so we should focus here on what is desirable for the UX.

[khangmach]

Updated the mockups to remove those blank blocks

[jtamiace]

I was worried at first that loading 16 at a time was too low of a number and might create a choppy or laggy experience on scroll, but if loading a few items at a time is fast enough, the continuous scroll experience could run smoothly.

Still, wondering if we can leave number of items per page flexible to whichever dev works on this, to test the limits on how many items actually creates a lag in load time?

[khangmach]

Sounds good. I'll leave the number of items loaded to whatever's dev-tested as optimal

[rtibbles]

Fixed in #8302