Django 1.3 introduces staticfiles which will raise an ImproperlyConfigured exception if you request a static file when settings.DEBUG is false. This avoids wasting time serving static media files but it would be nice if we could find an approach which would still allow us to check file existance.
The very first URL requested will cause a big (2+MB) memory delta as some delayed loading happens. To avoid skewing memory usage reports the client will make a single initial request before the actual monitored spidering.
This allows faster runs doing something like `django-admin.py crawlurls --no-parent /subpage/` and avoiding URLs which do not start with /subpage/
The CSV file will still display bytes but now the console output is friendlier.
This provides a simple --output-dir option which all plugins can use to save data as makes sense. This would still benefit from an easy way for plugins to have their own configuration when necessary. This introduces a set_output_dir() method on Plugin which subclasses may use to open log files or otherwise initialize their output system - see the guppy plugin for a simple example.
To avoid everything needing to be listed in __init__.py this introduces a simple change: more things are disabled by default and each plugin module has a PLUGIN attribute to simplify loading with --enable-plugins. Now enabled by default: time, pdb, urlconf