Demo website vs library

[iansmirlis]

I have some different results between the demo website and the library. In this particular case, the demo page gives better results. Does the demo page use some earlier version than the latest or makes use of some parameters other than the defaults?

Thanks

[gabriele-tomassetti]

Thanks for your feedback. The demo project is setup to use the same library and it does not do anything particular, it just sanitize content and download images. However I forgot to update the demo published on the website, that has remained on version 0.7.1.

Can you tell which URI looks better on the demo?

[iansmirlis]

Sure, the url is https://www.in.gr/1996/01/01/life/culture-live/cinema/kardia-toy-drakoy-h/

Demo site gives the correct content (as well as firefox reader view), however current version gives the copyright notice on the bottom of the page.

It's an edge case and I do not expect it to perform well in such page, I was just curious of why I have different results, I thought I was doing something wrong.

[gabriele-tomassetti]

I did find the reason of the difference. It happens because in this line the depth was changed from 3 to 5.

I am puzzled by the fact that your Firefox Reader View gets the previous result, since this is an upstream change and it is also quite old. In fact, on my personal installation of Firefox, if I click the button for Reader View I get the same result you get with the current library.

I am wary of changing back that value because Mozilla has a lot of testing data and we do not. Maybe we could add a configurable setting to allow changing that value.

[iansmirlis]

Oh ok, yes that makes sense because I use firefox debian 102.7.0esr version and not the current upstream.

You are right, there is no reason to change this, it will complicate things for no reason.

By the way, it seems that by lowering the CharThreshold parameter, it choose the correct article instead of copyright signature. I think you may close this issue.

[gabriele-tomassetti]

I tried to restore the old value and it does break some tests. However, I wonder if there is a way to guess a reasonable depth depending on the document. I will try to see if it possible, otherwise I will close the issue.

[iansmirlis]

I discovered the algorithm and this implementation only a couple of days ago and I still don't have good picture of the internal working.

However can this depth be a user parameter with the default value of the original implementation?

Is it affecting only speed if it's too high?

[gabriele-tomassetti]

I did add a setting to the Reader class to customize depth, it is called AncestorsDepth. First each node is scored depending on its content. Then part of the score of each node is added to the score of its ancestors depending on their relationships (i.e., close ancestors retain more value).
The lower the setting is, the lesser the value rise up the tree of nodes. Therefore is more likely that one distinct valuable node is chosen rather than one with many valuable descendants.