Browse files


  • Loading branch information...
pmyteh committed Jul 22, 2012
1 parent f0410d4 commit 6a40b8a159707f774c1cfc9d1616bbd81efe0b77
Showing with 2 additions and 3 deletions.
  1. +2 −3
@@ -4,11 +4,10 @@ phpWebCralwer
A web crawler in PHP.
Note that an additional file, CONFIG_db.php, is required. This sets the
-database server, name and password. Full details are commented in
+database server, name and password, as well as various other global options.
+An example file is included.
* Separate domain into its own db table.
* Set up per-domain timers, to allow local, not global, rate limit.
-* Move to libcurl for fetching -- restrict size/wrong MIME fetches.

0 comments on commit 6a40b8a

Please sign in to comment.