-
Notifications
You must be signed in to change notification settings - Fork 376
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MSC2291: Configuration to Control Crawling #2291
base: old_master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thank you! This sounds like a thing which might fit into the integrations spec when it exists, but could also fit nicely into the client-server spec (unlike widgets, though similar to widgets).
ps: merging develop -> the branch should fix the build error
"*": { | ||
"members": false | ||
}, | ||
"io.t2bot": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder whether it'd be easier for bot authors to parse io.t2bot
or io.t2bot*
here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that io.t2bot*
would imply that io.t2botfoobar
would use that key as well, and io.t2bot.*
is unclear whether or not a bot named simply io.t2bot
should use that key. But I'm largely indifferent to this issue, and anyone with strong opinions should give a good reason for one way or the other.
following the Java package naming convention. For example, the Voyager bot | ||
from t2bot.io could use the name `io.t2bot.voyager`. | ||
|
||
A new room state event `m.room.robots` is used to define what bots are allowed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if this could be applied to homeservers as well. I.e, for a homeserver to join a room, they must agree to keep room data confidential to the best of their ability. Homeserver owners would have to confirm that they do not mine data from rooms to be able to join rooms with such an option.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly, but that would probably be the subject for a different MSC. Though I think it would be hard to encode what policies a homeserver admin needs to agree with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alternative: policy lists as rooms
and recommend that bots leave when not allowed
Co-authored-by: Jonas Platte <jplatte@users.noreply.github.com>
- Otherwise, it will use the default value for that parameter. | ||
|
||
A bot may have multiple names that could be applicable to it. For example, if | ||
uhoreg.ca ran an instance of the Voyager bot, then the configuration for both |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How do these get defined? Do common ones get speced? Is it purely based on "others used this" or why not do this via a mxid? Basically how do I know what to look for when I build a bot? As having this for all bots individually would probably make this hard to use as admins
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The names would be using the Java package naming convention, but bot authors/admins would declare the name(s) that they use. So, for example, Travis could decide that his Voyager bot uses the io.t2bot.voyager
name. Your server stats bot could then use either both io.t2bot.voyager
and dev.nordeganken.serverstats
, if you think that its behaviour is close enough to the original behaviour, or just dev.nordeganken.serverstats
if you think that its behaviour is sufficiently different that it should no longer be considered as a Voyager bot.
This is similar to how web crawlers define their own User-agent
when checking robots.txt
.
Perhaps when we get extensible profiles, we can add something in there so that bots can declare which names the bot uses.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok yeah that makes sense. That just leaves me at how a) a room admin and b) new bot writers that arent as connected as we are get to know these keys :) Thats kinda one flaw I see here. While I see this MSC as necessary it is only as effective as the amount of known bot names/names bot filter for. For robots.txt it is pretty much solved by having lists for this and I am not sure how to solve this in spec considering the amount of time spec changes take. So maybe this should be part of appendix or even better a key in the bots entry for the "Try matrix now" page? (to make those keys somewhat discoverable)
rooms. This will be `false` if `messages` is `false`. Default: `true` if | ||
`m.room.history_visibility` is `world_readable`, and `false` otherwise. | ||
|
||
Bots may use other parameter names, but the names that are not listed in the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would the custom ones need to be named in a different format? Aka if I remember the "Java package naming convention" the above proposed ones do not follow this scheme. Or am I missing something? It feels inconsistent
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, we could use m.*
for the pre-defined keys. The format used here is consistent with what we do with room events, but aside from that, I'm fine with either way.
"io.t2bot": { | ||
"allow": false | ||
}, | ||
"io.t2bot.voyager": { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Combined with the other comment I made: Would this apply only to the voyager bot running on t2bot or would this apply to all voyager type bots? aka is this a type or a user? Especially as most voyager bots around do slightly different ways of crawling. So a room admin might be fine with the one by travis which only looks into future messages after join while the admin doesnt want to allow mine because my bot also looks into the history.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I mostly answered this in #2291 (comment) . Your bot could look up both io.t2bot.voyager
and dev.nordeganken.serverstats
, preferring dev.nordeganken.serverstats
. So if room admins could declare a config for just io.t2bot.voyager
, in which case all Voyager-type bots would use the same config. Or they could declare a config for both io.t2bot.voyager
and dev.nordeganken.serverstats
, in which case your bot would use dev.nordeganken.serverstats
for keys defined there and io.t2bot.voyager
for other keys, and other Voyager-type bots would only use the io.t2bot.voyager
config.
then the bot may not display any information about the room to users who are | ||
searching its directory, and may not store any information about the room | ||
other than its existence and its crawling preferences. The bot should also |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to best avoid displaying a room that isn't allowed, it would be nice if the m.room.robots
content would also be included in the /publicRooms
room directory response. Probably under the robots
key for a PublicRoomsChunk
.
Otherwise, a stateless app navigating the room directory has to make a request for each room to determine whether it's allowed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd be more in favour of adding it to the stripped state for the room, and exposing stripped state properly on /publicRooms
. The extensibility of PublicRoomsChunk
doesn't scale.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds great to me 👍
Stripped state spec docs for reference: https://spec.matrix.org/v1.7/client-server-api/#stripped-state
m.room.history_visibility
would be another good one to add but seems like that would fall better under a separate MSC.
to index the room may not want to join the room. However, bots may not be able | ||
to peek in rooms that its server is not already a part of until | ||
[MSC1777](https://github.com/matrix-org/matrix-doc/pull/1777) is fixed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The room summary API's from MSC3266 could cover the federated peeking niche.
Would just need to layer on the m.room.robots
info in a robots
key (same as the room directory)
(shout out to @turt2live for pointing MSC3266 out)
Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
then the bot may not display any information about the room to users who are | ||
searching its directory, and may not store any information about the room | ||
other than its existence and its crawling preferences. The bot should also |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the context of 2023 late october this problem came up again. It has become desirable to be able to opt out of aggregated room directory searches where you aggregate results from multiple room directories.
As a way of being able to use the robots event in this context a querry param could be used to ask only for rooms that allow them selfs to be returned in aggregated searches. Allowing the creation of a distinction like showing up on google search and being public on your website. But in this case it would be showing up in direct searches like those current gen clients do but being invisible to aggregated searches powered by spiders.
- `messages`: (boolean) whether the bot is allowed to index the room's | ||
messages. Default: `true` if `m.room.history_visibility` is | ||
`world_readable`, and `false` otherwise. | ||
- `log`: (boolean) whether the bot is allowed to display logs of the room to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is the difference between messages
and log
? Does there need to be two, or is one permission for messages in general fine?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The difference is that with just messages
, but not log
the bot can process the room's messages, but cannot display them to end users. For example, say that the bot is part of some room searching thing, and you ask it for rooms related to "cats". For rooms that just have messages
enabled, it can say "Here are some rooms that I think are related to cats". For rooms that have both messages
and log
enabled, it can say "Here are some rooms that I think are related to cats, and here are some messages from the room that are about cats". (I don't think it makes sense to have log
enabled, and messages
disabled.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(I don't think it makes sense to have
log
enabled, andmessages
disabled.)
(And in fact, the MSC does say that if messages
is false
, then log
is false
.)
Rendered