Skip to content
Heuristic text extraction from news sites in Python3
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.



Library that strips boilerplate HTML from news articles and performs heuristic analysis to determine the body of the article. Ranks text sections of the website by probability of being news content.

Currently uses for analysis:

  • Section Length
  • Section Position
  • Number of Anchors in a Section
  • Anchor Density in a Section
  • Word Count
  • Uppercase Word Count
  • Average Word Length
  • Average Sentence Length
  • Number of Sentences

This is a work in progress. I have manually tested it on several news websites, but extensive testing still needs to be performed.

Supports Python3

You can’t perform that action at this time.