Crawl a web site and count the number of links of specific types
Switch branches/tags
Nothing to show
Clone or download
Pull request Compare This branch is 1 commit ahead of Readz:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
resourcecrawler
tests
LICENSE
README.md
install
scrapy.cfg
setup.py
uninstall

README.md

LinkCrawler

Crawl a web site and count the number of links of specific types

--

Author(s):

Derek Frank <derek at readz dot com>

Install

Mac OS X (Yosemite)

Open Terminal /Applications/Utilities/Terminal.app

CMD-SPACE
terminal.app

Command line tools.

% xcode-select --install

Git makes retrieving files simple.

% git clone git@github.com:Readz/LinkCrawler.git || git clone https://github.com/Readz/LinkCrawler.git

Be sure to navigate within the base directory of the package.

% cd LinkCrawler

Run install script.

% ./install

Uninstall

Mac OS X (Yosemite)

Open Terminal and cd to your LinkCrawler directory.

Run uninstall script.

% ./uninstall

Example Usage

Print help.

% find_resources -h

Find all PDFs on cslibrary.stanford.edu

% find_resources http://cslibrary.stanford.edu

Find all images (PNG, JPEG, GIF, ...) on a single page.

% find_resources --dont-follow -c image http://cslibrary.stanford.edu/110 

Force include size information of resources.

% find_resources --deoptimize http://cslibrary.stanford.edu

Unless specified, the output CSV file will be located on the Desktop as "resources.csv"

% find_resources -o "Desktop/cslibrary-pdfs.csv" http://cslibrary.stanford.edu

Potential Platform Errors

OSX Python Cryptography source

MacPorts

% sudo port selfupdate
% sudo port upgrade outdated
% sudo port install openssl
% sudo env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/opt/local/lib" CFLAGS="-I/opt/local/include" pip install --upgrade --force-reinstall pyopenssl cryptography

Homebrew

% brew install openssl
% env ARCHFLAGS="-arch x86_64" LDFLAGS="-L/usr/local/opt/openssl/lib" CFLAGS="-I/usr/local/opt/openssl/include" pip install --upgrade --force-reinstall pyopenssl cryptography

Credits: