Only show world_readable rooms in the room directory

[MadLittleMods]

Only show world_readable rooms in the room directory

Happens to address part of #271 but made primarily as a follow-up to #239

Dev notes

Only fetch world_readable rooms from the room directory

Only 42% rooms on the matrix.org room directory are world_readable which means we will get pages of rooms that are half-empty most of the time if we just naively fetch 9 rooms at a time.

Ideally, we would be able to just add a filter directly to /publicRooms in order to only grab the world_readable rooms and still get full pages but the filter option doesn't allow us to slice by world_readable history visibility.

Instead, we have to paginate until we get a full grid of 9 rooms, then make a final /publicRooms request to backtrack to the exact continuation point so next page won't skip any rooms in between.

---

We had empty spaces in the grid before because some rooms in the room directory are private which we filtered out before. But that was a much more rare experience since only 2% of rooms were private .

Room creation -> Room directory lacks read-after-write consistency

Tracked by matrix-org/synapse#15526

This is why we added waitForResultsInHomeserverRoomDirectory(...). We first ran into this problem in #208 (comment)

Random

npm run test-e2e-interactive -- --grep "pagination is seamless"

Todo

• Paginate until we get a full grid of 9 rooms
• Add test to make sure pagination is seamless