Permalink
Switch branches/tags
Nothing to show
Commits on Mar 5, 2010
  1. added call to check got response from url before doing http request t…

    Mary Cook committed Mar 5, 2010
    …hat will block forever on an unresponsive host
  2. removed urls-saved passing; urls crawled in batches of 50; fixed a cr…

    Mary Cook committed Mar 5, 2010
    …asher caused by a malformed url
Commits on Feb 25, 2010
Commits on Feb 21, 2010
  1. unfruitful hosts avoided; speed-ups; files buffered before writing; b…

    Mary Cook committed Feb 21, 2010
    …etter url recognition;
Commits on Feb 17, 2010
  1. todo file

    Mary Cook committed Feb 17, 2010
  2. now saves place in crawl by writing state to files; way better commen…

    Mary Cook committed Feb 17, 2010
    …ts; added to list of uncrawlable urls
  3. added a few more comments

    Mary Cook committed Feb 17, 2010
  4. only crawls each url once; more uncrawlable urls excluded; outputs mp…

    Mary Cook committed Feb 17, 2010
    …3 urls to file
Commits on Feb 15, 2010
  1. first

    Mary Cook committed Feb 15, 2010
  2. first

    Mary Cook committed Feb 15, 2010