MSC2291: Configuration to Control Crawling #2291
Conversation
uhoreg
changed the title
There was a problem hiding this comment.
Looks good, thank you! This sounds like a thing which might fit into the integrations spec when it exists, but could also fit nicely into the client-server spec (unlike widgets, though similar to widgets).
ps: merging develop -> the branch should fix the build error
| "*": { | ||
| "members": false | ||
| }, | ||
| "io.t2bot": { |
There was a problem hiding this comment.
I wonder whether it'd be easier for bot authors to parse io.t2bot or io.t2bot* here.
There was a problem hiding this comment.
I think that io.t2bot* would imply that io.t2botfoobar would use that key as well, and io.t2bot.* is unclear whether or not a bot named simply io.t2bot should use that key. But I'm largely indifferent to this issue, and anyone with strong opinions should give a good reason for one way or the other.
| following the Java package naming convention. For example, the Voyager bot | ||
| from t2bot.io could use the name `io.t2bot.voyager`. | ||
|
|
||
| A new room state event `m.room.robots` is used to define what bots are allowed |
There was a problem hiding this comment.
I wonder if this could be applied to homeservers as well. I.e, for a homeserver to join a room, they must agree to keep room data confidential to the best of their ability. Homeserver owners would have to confirm that they do not mine data from rooms to be able to join rooms with such an option.
There was a problem hiding this comment.
Possibly, but that would probably be the subject for a different MSC. Though I think it would be hard to encode what policies a homeserver admin needs to agree with.
and recommend that bots leave when not allowed
Co-authored-by: Jonas Platte <jplatte@users.noreply.github.com>
turt2live
added
the
needs-implementation
| - Otherwise, it will use the default value for that parameter. | ||
|
|
||
| A bot may have multiple names that could be applicable to it. For example, if | ||
| uhoreg.ca ran an instance of the Voyager bot, then the configuration for both |
There was a problem hiding this comment.
How do these get defined? Do common ones get speced? Is it purely based on "others used this" or why not do this via a mxid? Basically how do I know what to look for when I build a bot? As having this for all bots individually would probably make this hard to use as admins
There was a problem hiding this comment.
The names would be using the Java package naming convention, but bot authors/admins would declare the name(s) that they use. So, for example, Travis could decide that his Voyager bot uses the io.t2bot.voyager name. Your server stats bot could then use either both io.t2bot.voyager and dev.nordeganken.serverstats, if you think that its behaviour is close enough to the original behaviour, or just dev.nordeganken.serverstats if you think that its behaviour is sufficiently different that it should no longer be considered as a Voyager bot.
This is similar to how web crawlers define their own User-agent when checking robots.txt.
Perhaps when we get extensible profiles, we can add something in there so that bots can declare which names the bot uses.
There was a problem hiding this comment.
ok yeah that makes sense. That just leaves me at how a) a room admin and b) new bot writers that arent as connected as we are get to know these keys :) Thats kinda one flaw I see here. While I see this MSC as necessary it is only as effective as the amount of known bot names/names bot filter for. For robots.txt it is pretty much solved by having lists for this and I am not sure how to solve this in spec considering the amount of time spec changes take. So maybe this should be part of appendix or even better a key in the bots entry for the "Try matrix now" page? (to make those keys somewhat discoverable)
| rooms. This will be `false` if `messages` is `false`. Default: `true` if | ||
| `m.room.history_visibility` is `world_readable`, and `false` otherwise. | ||
|
|
||
| Bots may use other parameter names, but the names that are not listed in the |
There was a problem hiding this comment.
Why would the custom ones need to be named in a different format? Aka if I remember the "Java package naming convention" the above proposed ones do not follow this scheme. Or am I missing something? It feels inconsistent
There was a problem hiding this comment.
Yeah, we could use m.* for the pre-defined keys. The format used here is consistent with what we do with room events, but aside from that, I'm fine with either way.
| "io.t2bot": { | ||
| "allow": false | ||
| }, | ||
| "io.t2bot.voyager": { |
There was a problem hiding this comment.
Combined with the other comment I made: Would this apply only to the voyager bot running on t2bot or would this apply to all voyager type bots? aka is this a type or a user? Especially as most voyager bots around do slightly different ways of crawling. So a room admin might be fine with the one by travis which only looks into future messages after join while the admin doesnt want to allow mine because my bot also looks into the history.
There was a problem hiding this comment.
I think I mostly answered this in #2291 (comment) . Your bot could look up both io.t2bot.voyager and dev.nordeganken.serverstats, preferring dev.nordeganken.serverstats. So if room admins could declare a config for just io.t2bot.voyager, in which case all Voyager-type bots would use the same config. Or they could declare a config for both io.t2bot.voyager and dev.nordeganken.serverstats, in which case your bot would use dev.nordeganken.serverstats for keys defined there and io.t2bot.voyager for other keys, and other Voyager-type bots would only use the io.t2bot.voyager config.
| then the bot may not display any information about the room to users who are | ||
| searching its directory, and may not store any information about the room | ||
| other than its existence and its crawling preferences. The bot should also |
There was a problem hiding this comment.
In order to best avoid displaying a room that isn't allowed, it would be nice if the m.room.robots content would also be included in the /publicRooms room directory response. Probably under the robots key for a PublicRoomsChunk.
Otherwise, a stateless app navigating the room directory has to make a request for each room to determine whether it's allowed.
There was a problem hiding this comment.
I'd be more in favour of adding it to the stripped state for the room, and exposing stripped state properly on /publicRooms. The extensibility of PublicRoomsChunk doesn't scale.
There was a problem hiding this comment.
Sounds great to me 👍
Stripped state spec docs for reference: https://spec.matrix.org/v1.7/client-server-api/#stripped-state
m.room.history_visibility would be another good one to add but seems like that would fall better under a separate MSC.
| to index the room may not want to join the room. However, bots may not be able | ||
| to peek in rooms that its server is not already a part of until | ||
| [MSC1777](https://github.com/matrix-org/matrix-doc/pull/1777) is fixed. |
There was a problem hiding this comment.
The room summary API's from MSC3266 could cover the federated peeking niche.
Would just need to layer on the m.room.robots info in a robots key (same as the room directory)
(shout out to @turt2live for pointing MSC3266 out)
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
| then the bot may not display any information about the room to users who are | ||
| searching its directory, and may not store any information about the room | ||
| other than its existence and its crawling preferences. The bot should also |
There was a problem hiding this comment.
In the context of 2023 late october this problem came up again. It has become desirable to be able to opt out of aggregated room directory searches where you aggregate results from multiple room directories.
As a way of being able to use the robots event in this context a querry param could be used to ask only for rooms that allow them selfs to be returned in aggregated searches. Allowing the creation of a distinction like showing up on google search and being public on your website. But in this case it would be showing up in direct searches like those current gen clients do but being invisible to aggregated searches powered by spiders.
| - `messages`: (boolean) whether the bot is allowed to index the room's | ||
| messages. Default: `true` if `m.room.history_visibility` is | ||
| `world_readable`, and `false` otherwise. | ||
| - `log`: (boolean) whether the bot is allowed to display logs of the room to |
There was a problem hiding this comment.
What is the difference between messages and log? Does there need to be two, or is one permission for messages in general fine?
There was a problem hiding this comment.
The difference is that with just messages, but not log the bot can process the room's messages, but cannot display them to end users. For example, say that the bot is part of some room searching thing, and you ask it for rooms related to "cats". For rooms that just have messages enabled, it can say "Here are some rooms that I think are related to cats". For rooms that have both messages and log enabled, it can say "Here are some rooms that I think are related to cats, and here are some messages from the room that are about cats". (I don't think it makes sense to have log enabled, and messages disabled.)
There was a problem hiding this comment.
(I don't think it makes sense to have
logenabled, andmessagesdisabled.)
(And in fact, the MSC does say that if messages is false, then log is false.)
Rendered