Browse files

Allow for fetching limited range of webpages, to avoid massive files

  • Loading branch information...
1 parent f0e99df commit 52d95b712087e221818b42537f5c857c305b4b23 @pmyteh pmyteh committed Jul 22, 2012
Showing with 4 additions and 0 deletions.
  1. +4 −0 CONFIG_db_example.php
@@ -21,6 +21,10 @@
//list of domains starting, ending, and separated with :
+//Fetch only first part of each page, to avoid huge files?
+// If $fetchrangeonly=true, what range to fetch? Here, the first 100KB is specified.
// Set spider penetration depth. If 0 crawl only pages in database.

0 comments on commit 52d95b7

Please sign in to comment.