Collect urls from the Twitter status sample stream
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Script to scrape and visit all Twitter sample stream urls

Simple script to scrape urls from the Twitter sample stream and resolve the urls to normal urls in order to gather data for fun.


  • Move to the root directory (next to
  • Run ./
  • Place your app credentials in

Creates a data directory where it will rotate the sample stream into separate txt files.

Script to copy HEAD and to the host given as first argument.

Script that reads data/*.txt and creates data/same_name.clean with the resolved (follow redirects) urls of the given txt file. To keep the script from being considered a bot, I randomly sleep some.