Crawlers, Parsers and Extraction

Crawlers

news-please

• News web crawler, can also be used to access Common Crawl
• Original paper here
• Provides heuristics for news classification from other web pages

Parsers

newspaper

• News article parser by Lucas Ou-Yang
• No new pull requests have been approved in 14 months
• Extensive and recently updated usage documentation provided by John Bumgarner: Newspaper Usage Overview

newshound

• New project started by John Bumgarner, pseudo-maintainer of newspaper
• Not yet released

Extraction

cc-pyspark

• Simple examples for Common Crawl extraction using PySpark
• Includes examples for wordcount, domain counting

cosr-back

• Backend of Common Search, a deprecated Common Crawl search engine
• Includes html parser for PySpark