(adding) Pipeline / Filter - keyword(s) Filter

[emphasize]

Hey there,

while almost all news sites structure their sites thematically (and therefor broad thematic crawling is possible) or using the elasticsearch (??) or databases indirectly for that matter later on is it planned to add said pipeline-filter (that we can drop non-keyword articles on the run) in the future?

Or did I miss something. Have crawled through the docs but can't find anything in that regard.

Best wishes

[fhamborg]

Hi,

I'm sorry but such filters are currently not planned as doing this kind of topic or semantic filter is not trivial if you're planning to do that with high reliability. Hence, I see this as a task that should be addressed externally. However, if you're talking about filtering by checking if an article's text contains one or more specific tokens, that could be easily added as another step of the Pipeline. Please reopen the ticket if you're planning to implement that and I'll be happy to support you should you have any questions.

Best,
Felix

[emphasize]

I think I have to explain myself in more detail.

I've looked through the docs from referenced projects like newspaper. Newspaper3k has this feature called "Keyword extraction from text". Now, like the "Date Filter" - which one can setup in the config.cfg -, it should be doable to filter out those articles right away, e.g. pipeline filtering. Anyways, this type of filtering rely on the presence of such keywords within the site (that's what you called Tokens, right?)

Isn't it doable to solve this issue through a parser?

I'm far away being a (good) coder guy. i'm looking at this more from a user perspective. Given the sheer (and growing) amount of "information" that should be a well received method within crawlers. That said, I can't estimate the complexity of such task.

Grüße

[fhamborg]

Keyword extraction from text sounds to me more like extraction of terms that are representative for the document, e.g., maybe with a high TF-IDF score or similar term scoring methods. That is something else than "filtering", at least if I understand you correctly. If I understand you correctly, the pipeline filter you would like to add should check whether a single article's main text (or title?) contains at least one (or all?) terms defined in a new parameter in the config file?

Or are you specifically referring to the keywords extracted by newspaper and want to check against them?

[emphasize]

The keyword is a JSON attribute extracted from the sites article like "authors", "publish_date" or "text"
Direct quote from newspaper3k docs (example site extracted)
`>>> article.keywords

['New Years', 'resolution', ...]`

Adding a parameter in the config file would allow to filter articles containing the keyword 'New Years' (for example), like the "Date filter" does. Other articles would be dropped in the pipeline. That's one (lean) way, BUT relies on the site containing this keywords.

Another way is to do the task through a/the parser (to filter atricles containing a given word), i presume. But that can be horribly wrong, or too complex. Or otherwise.

[fhamborg]

According to https://github.com/codelucas/newspaper the keywords are terms that are extracted when invoking article.nlp() so they don't need to be contained in the website but are generated by the newspaper library. I will close this issue, since we don't want any extractor-specific dependencies but keep news-please as general as possible. So far, I only know of newspaper extracting such "keywords".

I think what is more beneficial as to the goals of news-please would be to add a simple term filter as I described above.

Anyways, if you want to add such a feature (I guess it's roughly 20-40 SLOC that need to be added), you're more than welcome to contribute and I'll be happy to help you doing that.

[jmy48]

How would one begin to implement this keyword filtering step in the pipeline? I simply want to not save an article to disk if it doesn't contain a keyword in a set of keywords. Can I just stuff an if-statement somewhere before it saves?

[fhamborg]

See #101