Filter actions

[aledeg]

We have a recurring request, the need to filter articles when they are retrieved by FreshRSS. I recently went through all the issues and created a new tag for that matter (Article filter). At the moment, there is 13 open issues related to that. See here.

@Alkarex started to address those issues with the introduction of the Filter actions on the feed level. It's great but I find it limited and not very practical. Let me explain why.

I find it limited, because you can only match the current feed and it has only one action. I am aware that it was only a starter but I cannot foresee how we can improve it.
I find it not very practical, because if I need to make the same action on different feeds, I need to copy and paste over and over. When we will have more actions, we will need to copy and paste on that action as well.

I've been thinking about that for a while now because that's something I need. For instance, the Reddit Image extension I wrote could be transformed to be a filter action with the system I have in mind.

For me, we shouldn't have the filter actions on the feed level. Actually, I think the filter actions shouldn't be on any level but their own. They should be stored separately so we can create a filter action on a feed on day and decide the next day that we want to expand it to other feeds or categories without changing everything.

I think we should have a filter action chain composed by filter action link. This chain will be created once by reading the configuration and it will be use when inserting new articles. The chain will go through every link until one triggers its action.
For this to work, each link will have a matching rule and an action rule. The matching rule will be a combination of matching condition (ex: title contains xxx, category is xxx, ...). The action rule will be a combination of action (ex: mark as read, tag as xxx, highlight word xxx, ...)
With a system like that, it becomes easy to add new matching rules and new actions. We can start small to mimic what @Alkarex have done, one matching rule for the feed, one matching rule for the content and one mark as read action.
For the filter chain configuration, I can see something that looks like the filter in the webmail (Gmail, ...) that allows to describe every link one by one.
For the storage, I don't know if we need to write that in the database or in the configuration file.

I think that if we do this right, we can add a lot of flexibility in this process and I can even see me changing some part of my extension to fit into that filter action chain.

Let me know what you think!
Let me know if I need to add some clarifications.

[Alkarex]

Thanks @aledeg

My short-term roadmap (if possible for 1.18.0) is so far only covering automatically marking as read, and automatically applying labels:

1. Complete the work at feed level (one main feature that I want is missing, based on articles that are on the RSS server or not) Improved feed action filters #3303
2. Make default feed settings at category and global level (e.g. default filters to reduce copy-paste)
3. Make similar rules but applicable at category and global level (note: we already have a f:1,2,3 search filter, meant to limit to certain feeds Filter by multiple feed IDs #2892 )
4. Make a new configuration page for each label (just like for categories) (some help would be very appreciated here!)
5. Introduce new filters for auto-labelling (using point 4)
6. Introduce more automatisms at tag level, e.g. to automatically un-label.

I have wanted those features for a long time, very much along the lines of what you wrote @aledeg , and I have thought a lot about how they could actually be done (I thought of e.g. implementing a simple language to express actions, of various UIs with dynamic choices, etc. but discarded those options at this stage). I have come to the conclusion that in order not to be too complicated to use, we should (also) have some quite basic approaches (first). This does not exclude having more powerful approaches at higher level. I do believe that the plan above will cover a good part of the needs with reasonable efforts, relative ease of use, and good performance.

Step 1 is (just) a set of basic features, which can also be reused, just like when I introduced #2892
I do keep in mind more powerful wishes even when implementing those simpler options at the moment (see e.g. my comment #2398 (comment) )

In other words, I would very much support a global "filter/actions" as you propose, but would best like to land the simpler features first.

To faster reach this goal, having some help for step 4 (configuration page for the individual labels) in particular would be very much appreciated.

To prepare for steps 2 and 3, it would also be nice to do in advance some UI reorganisation of the options that can be at global/category/feed level , so help welcome there as well. See for instance:

I would like to have default settings for categories at global level, and default settings for feeds at category level.

I hope you will like this plan, @aledeg

[aklys]

I'm not sure if i should just add to this thread or start a new one so I'll just post here to begin with. With the filter/actions is there a reason it's gone focussed on mosty a singular action of being marked as read. In addition to global and category filters, has there been the though about making the filter be one of two options. a series of search functions or a regex statement associated with an action selected from a list. Such options may include "Mark as Read", "Mark as Unread", "Mark as Favourite", "Delete Article", "Apply Label: Blah".

While I appreciate this isn't probably easy to create to begin with the modular approach to this would enable future actions to be more easily implemented as the system gets more advanced. Happy to try and draw up some fake wireframes of what a config page might look like if it helps.

If this is already the plan and we are just progressing there ignore my idea :)

[Alkarex]

Regex filters #6706

[mtalexan]

In terms of the UI, you might take a look at what TTRSS does for some inspiration. It only has global action chains, so you have to define any/all filters in a single global list, pick which categories and/or feeds you want it to apply to, then specify a list of actions you want taken on the matches.

The major detriments of its design are that it's a breakdown of:

Filter
|- Rule List
|  |- Rule
|  |  |- Categories and Feeds
|  |- Rule
|  |  |- Categories and Feeds
|- Action Chain

And the pattern Rules have no AND/OR logic or parentheses. Resultantly you end up duplicating the same selection of categories and feeds for many different Rule patterns in the same Filter, and adding another feed that should be affected requires clicking thru a few screens on every Rule.

The other major detriment is that there's no way to see the results of what your Filter matched. The patterns can't just be copied into a search box because each pattern is in it's own sub-window that has to be individually opened, there's no button to send the pattern set to a search action, and there's no active search results match in the window itself.

---

EDIT

Screenshots:
A Filter for cooking-related terms with multiple Rule patterns trying to list the pattern, the field it matches against, and the feeds/categories it matches. Includes only a single toggle for AND or OR of all the Rules. Select, Add, and Delete buttons, and checkboxes on each rule to manage them. Clicking the rule opens it.

A Rule with the pattern, a drop down for the single field to match against, and a scrollable multi-select list of feeds and categories to match against. Patterns are only regex.

The action tab of the Filter Window, with Select, Add, and Delete buttons, and checkboxes on each rule to manage them. Clicking the action opens it. Actual selection opens a new pop up with a single drop down of possible actions and an OK and Cancel button. The few actions that are conflicting can't be selected for the same Filter.

TTRSS documentation is very clear that all actions from all Filters are effectively applied in parallel, so any conflicting decisions will result in race conditions on what the final result is. They don't do more than the most basic checks for conflicting actions in a single Filter, and leave it up to the user to make sure they don't write race-y rules.

[mtalexan]

A good next goal might be to add a way to specify re-usable filters that can be referenced from labels or feeds. Something like the User Queries, but they would need to be in the PHP syntax instead of the per-database syntax used for Search and User Query. Being able to define a complex set of filters in one global location associated with a name, and then reference it from any number of per-feed or label filters as something like psearch:'my special name' would be excellent. I'm currently doing the equivalent via a text document on my computer, and when I need to use the filter in a new place I copy-paste it from my text document into the set of existing filter patterns. And if I need to update one of these, I currently have to go track down what parts of the text in the tiny text boxes across many different feeds and labels I need to replace, and copy-paste into it.

[Alkarex]

We already support search:MyQuery or S:2
See https://freshrss.github.io/FreshRSS/en/users/10_filter.html#with-the-search-field
At the moment, when used in filter actions, those expressions are automatically replaced by what they are referring to, but we could change that.

[Alkarex]

And they can also be combined: search:MyQuery1 (!search:MyQuery2 OR search:MyQuery3)

[mtalexan]

They are substitited into label and feed filters, not just Searches?
When I saw that the saved User Queries included a list of feeds, and heard about the difference between label and feed filters vs database searches I assumed they couldn't be substituted into label and feed filters. If they can then that's perfect.

[mtalexan]

Do the feed IDs picked as part of the User Query get included in what's substituted in? For labels that would be convenient, but for feed filters it would be redundant since feed filters are already limited to the one feed.

[Alkarex]

Only the query string is used when included elsewhere, so no feed IDs