Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Pure python script that takes user query and summarizes news related to it.
Python
tree: 5ca82ffb70

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
GoogleNews
Summarize
TextRank
fetch_url
readability
Extract_News_Summary.py
Extract_News_Summary_TextRank.py
README

README

*************************************************************************
*       README file for Summarizing News Articles with user Query       *
*************************************************************************

REQUIREMENTS: a. Python and it's standard libraries
              b. NLTK along with the "punkt sentence tokenizer" and "english-pickle"
              c. Python lxml-library


The sub-folder contains the following MAIN scripts of the same name:

*************
*  SCRIPT 1 *
*************
GoogleNews.py - An independent script that can fetch the desired number of URLs for a particular search query

USAGE: The script consists of the following function:
        search(query, number of links)

query = Is the desired query seperated by '+' 

eg: If your desired search is "Mayawati BSP election" then the query string should be "Mayawati+BSP+election"



*************
*  SCRIPT 2 *
*************
readability.py - Independent script that extracts relevant text from a URL

USAGE: The script consists of uses a lot of support scripts for cleaning, housekeeping, debugging and logging. The "Document" class in the mentioned script isuseful for the task of extracting relavant text. Following is the way to use it in python terminal

>> from readability import Document
>> html = urllib2.urlopen(INSERT_URL_HERE).read()
>> article = Document(html).summary()
        

OUTPUT: Relevant Text. (Uses the algorithm by the arc90 readability project. Check my other github repo for the same)

*************
*  SCRIPT 3 *
*************
summarize.py - (Summarize folder) Independent script that creates the summary using simple analysis for sentences.

USAGE: Following is the way to use it in a python terminal
>> import summarize
>> ss = summarize.SimpleSummarizer()
>> summary = ss.summarize(INSERT_TEXT_TO_BE_SUMMARIZED_HERE,NUMBER_OF_LINES_FOR_SUMMARY)

OUTPUT: Summarized text.

*************
*  SCRIPT 4 *
*************
Summary.py - (TextRank folder) Independent script that creates the summary using TextRank for sentences.

USAGE: Following is the way to use it in a python terminal
>> import Summary
>> result = Summary.textrank(article)

OUTPUT: Summarized text.


*************
*  SCRIPT 5 *
*************
Extract_News_Summary.py - A script that uses all the above scripts

INPUT: Query and number of news articles to be scraped

USAGE: On the terminal do the following

python Extract_News_Summary.py <Number of links> <Query>

example: python Extract_News_Summary.py 50 India Pakistan Cricket

*************
*  SCRIPT 6 *
*************
Extract_News_Summary_TextRank.py - A script that generates summary of articles using the TextRank method

INPUT: Query and number of news articles to be scraped

USAGE: On the terminal do the following

python Extract_News_Summary.py <Number of links> <Query>

example: python Extract_News_Summary.py 50 India Pakistan Cricket


TODO: 1. Get urllib exception handling 
      2. Unicode-ASCII conversion is weakly handled
      3. If possible better scraping
      4. More complex Summarization algorithms
      5. urllib2 fetching is slow

Something went wrong with that request. Please try again.