New Filters System Specifications

[mbaruh]

This issue documents the planned changes to the filters extension. This needs to be done alongside python-discord/site#479.
Minor details may still change. This issue specifies the full extent of the desired changes, but this can be split into several issues and made into a project.

The changes allow for granular control over the behavior of individual filters, and increase the scope of it. All aspects of the extension, including automatic rules, will be controlled from a single entry-point.

Filter Settings

Each added filter has settings which dictate what happens if a given message matches it. The settings are:

• Does it ping?
  • Which roles? everyone/here/mods/something else
• Does it delete?
• Does it filter reactions?
• Does it filter usernames and nicknames?
• Does it catch DM's?
• Does it ping for DM's?
• Does it DM the user about the rule? (similarly to what happens for invite links / everyone pings)
  • A secondary configuration for the message sent
• Do the filters apply only in specific channels / categories
• Which roles can bypass the filter?
• Does it apply an infraction? warn / mute / ban
• For domains: be able to choose whether it should be exact, or apply for subdomains as well (this should close Filtering - New blacklist type that catch all subdomains of domain #1276)

Each filter additionally has a description, as they already do.

Each list type has default settings, while each individual filter can override any setting.

Filter Types

The current filters are split between four types: guild_invite, file_format, domain_name, filter_token. Each of those is split into a blacklist and a whitelist.

Renaming

The names should be simpler and more memorable: invites, formats, domains, tokens.

Whitelisting and Blacklisting

Not every filter type needs to have both a whitelist and a blacklist. Each filter type has its own method of deciding whether action should be taken, taking the whitelist, the blacklist, or both into account.

A blacklist or a whitelist should be created before anything can be added to it.

Invites

The filters should save the invite which was used to add a guild to the list, unlike the current state in which only the ID is saved.

Domains

Domains should support path patterns, e.g *.pythondiscord.com/*/ric*rolls.

New Type: Custom Functions

Each filter type has its own method of checking whether a specific filter should be applied. There should be a list which specifies special filters, each with their own way of deciding whether they apply to a given message. Examples of such filters: the token remover, webhook remover, and any of the antispam rules. When an item can and should be added to this list will be clearer in the Implementation Details section.

UI And Utilities

Search

There should be an option to present all filters in a specific list which match a given input.
For example, if we have the tokens a\db and 123, and we call !bl search tokens a4b, we will be presented with a list which contains the filter a\db. The listed filters should show each filter's ID (explained below).

Aliasing

There should alternative ways to phrase the same command:
!bl add tokens <token> -> !tokens deny <token>
!al add invites <invite> -> !invites allow <invite>
!bl list domains -> !domains list bl

find should be an alias of search

Reaction Menus

There should be reaction menus to go from !bl / !al, to one of the option: list, search, add, remove, which in turn should allow choosing one the possible filter lists (if there is no tokens whitelist it shouldn't be given as an option).

Similarly, following the aliasing section, specifying one of the list types as a top-level command should allow choosing one of list, search, allow, deny (if there is no whitelist, allow isn't an option).

Editing

As mentioned in the Settings section, each individual filter should have a way to override the default settings, including the pattern or the name of the filter which is used to dictate when it applies to a message. The only non-modifiable field of a filter is its ID (explained below).

Identification

Each filter should have its own ID, which is unique for a specific list. There should be a way to edit and remove a filter via that ID.

Additionally, it should be possible to bring up an embed with a particular filter's ID, pattern or name, and the values of all of its settings, via either the ID or the pattern / name.

Alerting

As different filters can have different affects on a message, it should be explored whether it's possible to present in the alert all tokens which triggered unique filters (meaning, we don't have to show all tokens that triggered a specific filter).

User Ignore List

There should be a command which makes a specific filter temporarily not apply to a specified user. This is the same feature as in #551, except a redis cache might be enough.

Implementation Details

OOP-Based Approach

• There should be a Settings dataclass that implements a binary operation to join settings together. This will allow us to decide all the actions that should be taken for a specific message. For example, if one filter that is matched says the message should be deleted, and another says the moderators should be pinged, they should be joined together, and then actioned after going over all filters. There should also be a method to apply overrides to a Settings object, returning a new Settings object.

• There should be a base Filter class with a settings attribute, that is then subclassed for each filter type to code a predicate to decide whether the filter should be applied (along the lines of async def matches(message: Message) -> bool). This will also allow the custom filters to be subclasses of Filter.

• Lastly, there should be a base FilterList class, that is then subclassed to specify which Filters class it uses, how it loads and initializes its filters, and what it does with its blacklist and/or whitelist. Each list loads its filters from the database, transforms each into an object of a Filter subclass, and stores it in the list corresponding to the blacklist or whitelist.

Extensibility and Modularity

I don't claim to foresee every future need or use case. For that reason, during development special attention should be made to extensibility (this also applies to the site issue). Most parts should be modular, with several dispatching methods. A good indication of a hard-to-extend implementation would be if a fragmented diff, scattered across a file, is necessary to say add an additional type of filter, or an additional effect a filter can have.

Furthermore, a module, a file, or a class should have a clear documentation on what parts need to be looked at and added / changed in order to add or remove an additional filter setting or filter type.

Each FilterList class should be specified in its own file, within a dedicated module, and all of them should be dynamically loaded in the cog from said module.

Similarly, each custom filter should be in its own file, within a dedicated module. The filter within the custom filters list will then store the name of the class (similarly to how tokens store a regex), which will then be dynamically loaded from said module (as a reminder, each FilterList subclass can override how it loads its filters).

Related Issues

Additionally to #1276 and #551 mentioned above, this specification should allow to gracefully handle #1018, #1421.

#665, #998, #1379, #1522 are additional issues related to the functionality of one or more filters that can be handled as a part of this rewrite, but technically speaking are out of scope for this issue.

[MarkKoz]

Do the filters apply only in specific channels / categories

What about filtering by roles? The obvious use case is to exclude staff members from some filters.

Reaction Menus

While I'm not the target user for this feature, I'm not sold on the necessity of it.

it should be explored whether it's possible to present in the alert all tokens which triggered unique filters (meaning, we don't have to show all tokens that triggered a specific filter).

Can you rephrase this? I don't understand what the difference is between the sentence outside the parenthesis and the one inside the parenthesis.

User Ignore List

What kind of granularity is needed? Specific filter ID(a), filter list type(s), or all filters?

There should be a base Filter class with a settings attribute, that is then subclassed for each filter type to code a predicate to decide whether the filter should be applied (along the lines of async def matches(message: Message) -> bool). This will also allow the custom filters to be subclasses of Filter.

Is defining subclasses really necessary? I don't see why the predicates would need to be stateful. It could just as easily be done through something like

class Filter(NamedTuple):
    settings: Settings
    predicate: Callable

class FilterRegistry:  # For the sake of example...
    def add_filter(self, settings: Settings, predicate: Callable):
        self.filters.append(Filter(settings, predicate))

[mbaruh]

What about filtering by roles? The obvious use case is to exclude staff members from some filters.

You're right; edited.

While I'm not the target user for this feature, I'm not sold on the necessity of it.

The commands are kinda long and hard to remember, so what ends up happening is that someone will use !bl, and then they see the available commands, and then they use say !bl list, because we don't remember what the names of the lists are, and only after seeing the list names we'll call the full command !bl list filter_token (for example). So that's the same routine of three commands being called just to do the one operation. If there was a way to transition from whatever starting point you remember to the desired output that would be much better. As I'm saying all this I realize that slash commands would do very well here, but I'm not sure we want to go that route yet?

Can you rephrase this? I don't understand what the difference is between the sentence outside the parenthesis and the one inside the parenthesis.

Both sentences say the same thing, I just tried to make it easier to understand in case someone didn't get the phrasing in the first sentence. The point is that different filters trigger different actions, so unlike now, several might need to be shown to understand the cause for the full set of actions taken. At the same time, we don't want to show the same filter twice.

Is defining subclasses really necessary? I don't see why the predicates would need to be stateful. It could just as easily be done through something like <...>

The issue is that it's not possible for the list of custom filters, because each such custom filter will have its own predicate, which can be arbitrarily complex. So I'd rather if those classes contained their predicates, and then load those filters dynamically from a module, instead of the filter list containing all of the predicates and then dispatching them.
While it seems possible to do what you're showing for other list types with non-changing predicates, I'm wondering if it's better to just have the same design for all of them.

It's also entirely possible for a predicate to depend on the settings.

[mbaruh]

Missed this one

What kind of granularity is needed? Specific filter ID(a), filter list type(s), or all filters?

Whatever is needed to uniquely identify a filter. By how the site schema looks right now (not sure if Akarys updated it after opening the issue, we continued discussing it since), it seems that each filter will have a unique ID. So a filter ID - user ID's pairing should be enough.