Parse discovered room directories to discover new rooms

[turt2live]

This requires a lot of backend work and possibly some rate limit handling magic.

Could represent using something like this: https://bl.ocks.org/micahstubbs/f32a742947f259d12eb999d418a767fb

[turt2live]

This should only parse directories it knows about based on members in the rooms it finds. When it finds a room is no longer linked to the directory, it should remove it from the graph. This should be done as a scheduled task to reduce watcher-style scanning of directories.

[turt2live]

This needs more thought and discussion. Recent events in some of the Riot/Matrix channels have led to some community concern about bots that parse directories, joining the rooms.

Here's some relevant discussion points:

• Set default federate by settings matrix-org/synapse#2199 (comment)
• Add require_auth_for_room_directory option matrix-org/synapse#2421
• Option to lock /publicRooms access only to local users matrix-org/synapse#2419

I'm interested to hear feedback from people in the community: How should voyager deal with directories, if at all? Some options might be:

1. Join the rooms (seems to be discouraged by the community so far)
2. List the rooms on the graph, but don't join (add/remove nodes as they appear/disappear from directories)
3. Close this ticket without implementing the feature.
4. Some other way?

[turt2live]

A brief chat with a couple users in #offtopic yielded the following:

Voyager joining rooms from /publicRooms is okay, if:

• It does not leave without being asked to (or kicked, etc. It should stick around)
• The information it collects is actually used (ie: displayed in the graph)
• Is not invasive (doesn't join and start spamming rooms or causing significant load on the server)
• Has a way for Voyager to be removed from all rooms on a server's directory easily (effectively a server ban)

I feel this is a reasonable expectation of the bot, and am still curious to see what other people think. Is the above okay?

[turt2live]

Some related discussion in another related project brought up an interesting point that maybe the bot should reach out to the administrators of the server first. This would be hard to identify, given there's no spec, but maybe if an endpoint were added then Voyager could use that.

More thought is required on this issue.