Code for Scraping the Altiscale/Internet Archive .gov Data - Political Methodologist Article
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
Climate_UniqueCaptures.pig
ExtractCounts_keywords.pig
README.md
UDFs.py
trend2.pig

README.md

.govDataAnalysis

Code for Scraping the Altiscale/Internet Archive .gov Data for Political Science Research

Documents in this repository include scripts for scraping the .gov data for two main purposes:

1)	tracing the history of an entire webpage and either outputting the full text of those pages or outputting keywords from those pages
  
2)	tracing the entire history of a given set of terms, and then either outputting the full text of those pages or outputting keyword counts from those pages
  
  a.	also possible to sort and categorize by URL group (the URL roots of interest)