Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Commits on Aug 5, 2015
  1. Merge pull request #39 from zmaillard/json_std

    Added json Standard Module to Support Django 1.7+
  2. @sagebrushgis
Commits on Jun 29, 2015
  1. Merge pull request #25 from fabrique/time_data_to_csv

    [crawler] Write out request times to a csv if output_dir is specified
Commits on Sep 6, 2013
  1. Merge pull request #26 from fabrique/skip_static

    [crawler] Skip static/media paths as it doesn't makes sense to crawl them
  2. Merge pull request #31 from ernop/master

    fix an error when making fixtures
  3. Merge pull request #33 from jpic/patch-1

    Enable twil.go('urlname_without_args')
  4. Merge pull request #34 from millioner/master

    Django 1.4 load_template_source import error
  5. Merge pull request #35 from zenweasel/master

    Fix minor doc typo
  6. Merge pull request #38 from mlissner/patch-1

    Update README
  7. @mlissner

    Update README

    mlissner committed
    Fixes the link to the docs.
Commits on Mar 3, 2013
  1. @zenweasel

    Fixed typo, closes issue #32

    zenweasel committed
Commits on Feb 26, 2013
  1. @millioner
Commits on Nov 19, 2012
  1. @jpic

    Bugfix: twil.go('urlname_without_args') now works.

    jpic committed
    Before, this had to be used:
        twil.go('urlname_without_args', args=[])
    Now this is supported:
Commits on Jun 7, 2012
  1. @ernop

    fixes #28

    ernop committed
Commits on Jan 3, 2012
  1. @tino
  2. @tino
Commits on Dec 28, 2011
  1. Merge pull request #23 from michelts/master

    Make quicktest command compatible with django 1.3
Commits on Dec 16, 2011
  1. Make quicktest command compatible with django1.3 by changing the impo…

    Michel Sabchuk committed
    …rts on the module.
Commits on Apr 23, 2011
Commits on Mar 14, 2011
Commits on Feb 10, 2011
  1. @saltycrane
Commits on Nov 4, 2010
  1. Added mailing list info.

Commits on Oct 3, 2010
  1. @j2a
Commits on Sep 9, 2010
  1. @acdha

    Crawler: init request for accurate memory tracking

    acdha committed
    The very first URL requested will cause a big (2+MB) memory delta as some delayed loading happens. To avoid skewing memory usage reports the client will make a single initial request before the actual monitored spidering.
  2. @acdha

    Crawler: added a no-parent option to avoid ascending

    acdha committed
    This allows faster runs doing something like ` crawlurls --no-parent /subpage/` and avoiding URLs which do not start with /subpage/
  3. @acdha

    Crawler: adjusted log level for link crawling

    acdha committed
    This avoids tons of console output by default
  4. @acdha
  5. @acdha

    Crawler: query_count log file

    acdha committed
  6. @acdha

    Crawler: guppy plugin now uses human-readable sizes

    acdha committed
    The CSV file will still display bytes but now the console output is
  7. @acdha

    Crawler: uniform system for saving output

    acdha committed
    This provides a simple --output-dir option which all plugins can use to save data as makes sense. This would still benefit from an easy way for plugins to have their own configuration when necessary.
    This introduces a set_output_dir() method on Plugin which subclasses may use to open log files or otherwise initialize their output system - see the guppy plugin for a simple example.
  8. @acdha

    Crawler: guppy plugin simplification

    acdha committed
    Since we now only load the guppy plugin when the user requested it, it's
    better just to toss an import error if we can't load the guppy module.
  9. @acdha

    Crawler: better plugin activation mechanism

    acdha committed
    To avoid everything needing to be listed in this introduces a simple change: more things are disabled by default and each plugin module has a PLUGIN attribute to simplify loading with --enable-plugins.
    Now enabled by default: time, pdb, urlconf
Commits on Sep 8, 2010
  1. @acdha

    Whitespace cleanup

    acdha committed
Something went wrong with that request. Please try again.