These flags allow you to change the behavior of Crawler. Check out how to use them in the Cookbook.
-d <sec>, --delay <sec>
Use a delay in between page fetchs so we don't overwhelm the remote server. Value in seconds.
Default: 1 second
-i <regex>, --ignore <regex>
Ignore pages that match a specific pattern.
Default: None