Site-specific rules

[n1k0]

Context: #75 (comment)

The universal approach of Readability.js is admittedly convenient, though doesn't work for every website, where markup and {js|css}-driven behavior can easily get in the way trying to extract meaningful contents. That becomes a serious issue in the case of a popular website like Fastcompany…

We should start considering adding per-site rules, in order to guarantee users to access readable contents for mainstream websites. A very simplistic approach of how a configuration could look like:

// ./rules/fastcompany.com.js
ReadabilityExt.registerRule({
  remove: [
    "p > a.people-page img",
    // … moar selectors
  ]
});

Obviously, that means JSDOMParser to support querySelectorAll and friends, though I think there's work currently ongoing on this front.

Of course we should think a little harder of how the whole thing would work, which options would make sense, the possible impact on performance and so on.

We'd have to maintain per-site rules over time as websites' markup change… this could quickly become tedious, so we should restrain a very short list of supported websites. We could also rely on a community effort, though that would mean a super easy way of doing so.

Note: This is quite possibly out of scope for 38.

Thoughts?

[leibovic]

This is interesting. Given the case of removing elements, I wonder if we could get away with doing something with CSS. If the problem is that we keep more elements than we want, we could add CSS rules in aboutReader.css (in the Firefox use case) to just hide these extra elements.

However, the case where we want to keep elements we would otherwise remove would be harder (impossible?) to implement with this approach. With this CSS-based approach to work, I think we would have to err on the side of not removing so many things, but that becomes tricky, since we obviously don't want these kinds of rules for all sites. Maybe we could not remove as many elements, but default some of the questionable ones to be hidden?

From the performance side of things, as well as actually getting this to work, I think it might make sense to look through the different steps of the algorithm, and have rules that apply to those different steps. For example, when we're iterating through removing "unlikely candidates", maybe we can check a rule to make sure we don't remove something for a certain site.

[n1k0]

Given the case of removing elements, I wonder if we could get away with doing something with CSS

Yeah, that would make total sense, indeed. The only thing is that CSS isn't unit-testable at all (don't get me started on visual regression/imagediff tooling); so maintaining these CSS rules, eg. checking for regressions or unwanted side-effects will be more painful because visually tested by humans… I wish we could stick to pure automated testing, which is what programmatic site-specific rules could much more easily bring.

[maxnordlund]

This could be used to whitelist/blacklist certain sites as well, like *.blogspot.com -> #108. Not site specific, but engine ditto, is to special case popular publishing tools like wordpress and mediawiki. Not sure how to detect that reliably though, but maybe a combination of HTTP response headers and meta tags will suffice.

Example meta tag from wikipedia:

<meta name="generator" content="MediaWiki 1.26wmf14" />

[SpyC0der77]

The only thing is that CSS isn't unit-testable at all (don't get me started on visual regression/imagediff tooling); so maintaining these CSS rules, eg. checking for regressions or unwanted side-effects will be more painful because visually tested by humans… I wish we could stick to pure automated testing, which is what programmatic site-specific rules could much more easily bring.

11 years later, AI can do this, maybe worth reconsidering

[SpyC0der77]

Will look into familiarizing myself with the codebase and maybe trying things out