Difference between Safari and SmartReader + FireFox output

[PeterHagen]

I have been trying and using SmartReader successfully in most cases. I ran into a difference in output between Safari and SmartReader (and Firefox).

When I parse this address https://tweakers.net/downloads/69820/libreoffice-2426.html with SmartReader (or FireFox), I don't get the article, but do get the comments:

Looking at the HTML, I don't see anything strange to it:

<div id="layout" class="content-grid ">
    <div id="contentArea">
        ...
        <div class="articleColumn wide">
            <div class="headingContent">
                <header class="title">
                    <h1><span data-cy="articleTitle">Software-update: LibreOffice 24.2.6</span></h1>
                </header>
            </div>
            <div class="articleContent">
                <div class="article largeWidth">
                    <p>
                        <img alt="LibreOffice logo (75 pix)" class="alignRight" height="79" src="https://tweakers.net/ext/i/1295963272.png" title="LibreOffice logo (75 pix)" width="65">
                        Onlangs is versie 24.8 van LibreOffice ....
                    </p>

But, when I open the same page with Safari, it does give the correct content, without the user comments:

What could be the reason for this difference, and is there a way to resolve this?

[PeterHagen]

Debugging the parsing, it seems that the comment will receive higher points then the document. Any suggestions how this could be avoided?

[gabriele-tomassetti]

Thanks for the bug report. If you are interested just in solving the issue for that website, you can use the method AddOptionToRegularExpression to boost the content part and devalue the comment part. You can add the id and class of the comment section to UnlikelyCandidates and Negative while adding the class and id of the main content to PossibleCandidates and Positive. I would start with adding the class reacties-inner (which includes the comments) to the UnlikelyCandidates and Negative regular expressions.

I do not know what Safari uses to implement the Reader mode-equivalent. They might have started with the same software, but maybe Safari's version of the readability engine has diverged from the one used by Firefox.

[PeterHagen]

I saw someone took the script sources from Safari and put them in a repository, if you are interested. As far as I could find online, Safari's implementation comes from the same readability libraries, but this could be old information.

I'll try your suggestion!

[PeterHagen]

RegularExpressions.UnlikelyCandidates with reacties-inner does do the job. So I can work around it, but this is then of course a custom solution.

[gabriele-tomassetti]

Thanks for showing me the repository. I do not think we can use any of it, since it has no open source license.
I am not sure we can improve our algoritm, but I will take a look in the new few days.
Is reacties typically used in place of comments in Dutch? If so, we can add it and several translations of the words comments in the to the regular expression for UnlikelyCandidates.

[PeterHagen]

Is reacties typically used in place of comments in Dutch?

Yeah, "reacties" is typically used for "comments".

If so, we can add it and several translations of the words comments in the to the regular expression for UnlikelyCandidates.

That would be perfect! I wasn't that far in dissecting the code to figure that out.

[gabriele-tomassetti]

I have added a fix to this issue. Let me know if you have anything else in mind, otherwise I will publish a new version of the library in the following days.

[PeterHagen]

For now this is the only situation I found. If I find any others (and hopefully resolve them myself), I'll let u know!