5c5202dba39a1e28d9321617be34ed6e7e83591f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/7dd2e87f8db18fade11428c834f9a1d6f05c1e5d 7dd2e87f8db18fade11428c834f9a1d6f05c1e5d 2008-07-22T16:47:31-07:00 2008-07-22T16:47:31-07:00 minor 611031c7db2993df1eafe2c01aafa914405cd1e9 Martin Matusiak numerodix@gmail.com 7ea823e2ff945f0c0b0fb9cbfb69f4915cf6e370 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5c5202dba39a1e28d9321617be34ed6e7e83591f 5c5202dba39a1e28d9321617be34ed6e7e83591f 2008-07-22T11:15:15-07:00 2008-07-22T11:15:15-07:00 mysql and sqlite disagree on insert or ignore format GAH! 155e5eb149d853b00a8628652294ec9c2994d1a0 Martin Matusiak numerodix@gmail.com a54e717c7eaeb6299f95e83af755cca62bd42c5c Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/7ea823e2ff945f0c0b0fb9cbfb69f4915cf6e370 7ea823e2ff945f0c0b0fb9cbfb69f4915cf6e370 2008-07-22T10:48:13-07:00 2008-07-22T10:48:13-07:00 allow cycling functions dfdb389a2b508c27bce2f68ba0c37032e21ecc5f Martin Matusiak numerodix@gmail.com ea74c11c0a735ead2a4179a7f00a8457c05c02a9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/a54e717c7eaeb6299f95e83af755cca62bd42c5c a54e717c7eaeb6299f95e83af755cca62bd42c5c 2008-07-22T08:28:24-07:00 2008-07-22T08:24:34-07:00 support mysql, sqlite 96f52c101f9391471199a8c9496718ee5efc5508 Martin Matusiak numerodix@gmail.com cadef459c2c181ace858dd35c901714ff5115cc7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/ea74c11c0a735ead2a4179a7f00a8457c05c02a9 ea74c11c0a735ead2a4179a7f00a8457c05c02a9 2008-07-22T08:27:51-07:00 2008-07-21T18:37:46-07:00 support cmd arg 23fde2db10a7a41944e63b9d24f4fdbbaf5adff1 Martin Matusiak numerodix@gmail.com 325e60d69a0e9b4533abb91af6d08c7ac38d21ba Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/cadef459c2c181ace858dd35c901714ff5115cc7 cadef459c2c181ace858dd35c901714ff5115cc7 2008-07-22T04:16:09-07:00 2008-07-22T04:16:09-07:00 mysql support 6625903333374078b802afe8576ac91c7b7866b4 Martin Matusiak numerodix@gmail.com 34a1192c241cc5da48c5f8b425e76ebf38dc36f2 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/325e60d69a0e9b4533abb91af6d08c7ac38d21ba 325e60d69a0e9b4533abb91af6d08c7ac38d21ba 2008-07-21T18:37:46-07:00 2008-07-21T18:37:46-07:00 support cmd arg 329766adad333f8e3c735d4ae6f53385bb00b7c8 Martin Matusiak numerodix@gmail.com 796bbefe973acd38f27f5014ede32ae3d7563857 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/34a1192c241cc5da48c5f8b425e76ebf38dc36f2 34a1192c241cc5da48c5f8b425e76ebf38dc36f2 2008-07-21T14:09:07-07:00 2008-07-21T14:09:07-07:00 formatting 25d77d00b8521210d8295bb803a4f3afd12bc26c Martin Matusiak numerodix@gmail.com 21fa8db8ddb95ac411da7dcfdd7f8523357b2a0d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/796bbefe973acd38f27f5014ede32ae3d7563857 796bbefe973acd38f27f5014ede32ae3d7563857 2008-07-21T13:02:10-07:00 2008-07-21T13:02:10-07:00 barebones 617c9b18bfea714ac1da2cee283ed69ed845f856 Martin Matusiak numerodix@gmail.com 109f7f53a85a50da068f33b956c9feb43a6a1865 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/21fa8db8ddb95ac411da7dcfdd7f8523357b2a0d 21fa8db8ddb95ac411da7dcfdd7f8523357b2a0d 2008-07-20T10:25:59-07:00 2008-07-20T10:22:22-07:00 perf 23fde2db10a7a41944e63b9d24f4fdbbaf5adff1 Martin Matusiak numerodix@gmail.com b5e4a4b530320579cbefa422456ad89876686020 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/109f7f53a85a50da068f33b956c9feb43a6a1865 109f7f53a85a50da068f33b956c9feb43a6a1865 2008-07-20T10:25:37-07:00 2008-07-20T10:25:37-07:00 consistency 69379d1c9ac696b76a9a28b2c6a816b3ada483db Martin Matusiak numerodix@gmail.com 2d4eb68132fc2310df83911695bc56f763d78f9a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b5e4a4b530320579cbefa422456ad89876686020 b5e4a4b530320579cbefa422456ad89876686020 2008-07-20T06:51:10-07:00 2008-07-20T06:51:10-07:00 comments 4fa44be7c944eeaf9d10a696294e36e24148192a Martin Matusiak numerodix@gmail.com 19f5f1bd571f96420f2c52b39735376dea673de4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2d4eb68132fc2310df83911695bc56f763d78f9a 2d4eb68132fc2310df83911695bc56f763d78f9a 2008-07-20T06:49:52-07:00 2008-07-20T06:49:52-07:00 move web save/restore to web module 6980540840ed62fef35da9e4a4e2e762a50ee244 Martin Matusiak numerodix@gmail.com 659ba2ca156192dc1a073619fd366ca8a69f53b9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/19f5f1bd571f96420f2c52b39735376dea673de4 19f5f1bd571f96420f2c52b39735376dea673de4 2008-07-20T05:15:39-07:00 2008-07-20T05:15:39-07:00 organize 6d4f35a8054bd8b1bdc8de927ce00f263225d2db Martin Matusiak numerodix@gmail.com bb46140e3e587706df3251ea794628b331b4d590 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/659ba2ca156192dc1a073619fd366ca8a69f53b9 659ba2ca156192dc1a073619fd366ca8a69f53b9 2008-07-20T04:37:01-07:00 2008-07-20T04:37:01-07:00 revert att names to pre rename for compat ead0c7143f48ba106d384cc03a6b742e7e9ae3e1 Martin Matusiak numerodix@gmail.com 1151f0ab0b360b19b9140f4a9862d88850299ff8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/bb46140e3e587706df3251ea794628b331b4d590 bb46140e3e587706df3251ea794628b331b4d590 2008-07-20T04:32:20-07:00 2008-07-20T04:32:20-07:00 add nodeid att to Node in preparation for db subclassing a2ed131a2a80f06b9b48c60fabff9be1be2e3b88 Martin Matusiak numerodix@gmail.com 950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1151f0ab0b360b19b9140f4a9862d88850299ff8 1151f0ab0b360b19b9140f4a9862d88850299ff8 2008-07-19T20:40:40-07:00 2008-07-19T20:40:40-07:00 api rewrite of web into small functional units for db subclassing 9da7c183ab11e2aecf85a2729640d3e9b948ee7b Martin Matusiak numerodix@gmail.com 62dd9e94779ca8fb259c69f29bdc737b156f3e16 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 2008-07-19T20:35:25-07:00 2008-07-19T20:35:08-07:00 prune unused gzip add header func 74c0fdec2fb0aeea2f377f8f19fad4aa4f404235 Martin Matusiak numerodix@gmail.com d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/62dd9e94779ca8fb259c69f29bdc737b156f3e16 62dd9e94779ca8fb259c69f29bdc737b156f3e16 2008-07-19T11:15:42-07:00 2008-07-19T11:15:42-07:00 make queue and wb instance variables 084e09bf07edf9f28098f2b2cb6761f7caf0d567 Martin Matusiak numerodix@gmail.com 554af5a465a020adf72903f094bd96987449fe10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 2008-07-19T09:08:58-07:00 2008-07-19T09:08:58-07:00 wrap spiderfetch methods in class, manage state variables better bb3b948e4308b08593c22a951e896962188458c1 Martin Matusiak numerodix@gmail.com 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/554af5a465a020adf72903f094bd96987449fe10 554af5a465a020adf72903f094bd96987449fe10 2008-07-17T14:48:23-07:00 2008-07-17T14:48:23-07:00 spider.py also supports local files, include that in the help d164fbf3356e354a72da509b1714d55b36373dce Martin Matusiak numerodix@gmail.com 5d2f989aee8ea812b52c812aef5149f69c036a10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 2008-07-17T11:37:59-07:00 2008-07-17T11:37:59-07:00 periodically save state 6f4782d744c4e5cc88be907d0857857a0e483923 Martin Matusiak numerodix@gmail.com 67359833ad7a2d8e2b71794902cc10ee77b43845 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5d2f989aee8ea812b52c812aef5149f69c036a10 5d2f989aee8ea812b52c812aef5149f69c036a10 2008-07-17T10:56:58-07:00 2008-07-17T10:56:58-07:00 safer serialization by writing to .partial files 97f8d3e84d92035e6d8a1779fbf2ba4019aa4c0c Martin Matusiak numerodix@gmail.com 73432359c7508fb7b74ef08e16f0457ea81536d2 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/67359833ad7a2d8e2b71794902cc10ee77b43845 67359833ad7a2d8e2b71794902cc10ee77b43845 2008-07-09T14:57:56-07:00 2008-07-09T14:57:56-07:00 formatting 3181f7bf905381eeaffbd76a0878fc900344883a Martin Matusiak numerodix@gmail.com 1ff1a04b833f4e01fa530612c80ae73cde49d9a6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/73432359c7508fb7b74ef08e16f0457ea81536d2 73432359c7508fb7b74ef08e16f0457ea81536d2 2008-07-08T18:37:32-07:00 2008-07-08T18:37:32-07:00 avoid division by zero 1b35ae6c0328a873ad37656e587c930f537de4c0 Martin Matusiak numerodix@gmail.com c6515b4b460cd2805b8086b1045e48651d643814 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1ff1a04b833f4e01fa530612c80ae73cde49d9a6 1ff1a04b833f4e01fa530612c80ae73cde49d9a6 2008-07-08T13:13:16-07:00 2008-07-08T13:13:16-07:00 fix naming bug related to -c ed03a81a9540e547ee3f8e56f7550013ecec0b76 Martin Matusiak numerodix@gmail.com d4f6f84c8ffb3dfe8076554afe3582856798a7a3 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c6515b4b460cd2805b8086b1045e48651d643814 c6515b4b460cd2805b8086b1045e48651d643814 2008-07-08T12:52:37-07:00 2008-07-08T12:52:37-07:00 add -t option to fetcher 41c4dca76eadcbac345723d8be55206a96617519 Martin Matusiak numerodix@gmail.com 1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d4f6f84c8ffb3dfe8076554afe3582856798a7a3 d4f6f84c8ffb3dfe8076554afe3582856798a7a3 2008-07-08T10:33:40-07:00 2008-07-08T10:33:40-07:00 file naming fix 21936e7dcc8add399c900fc7711caab2051e6b44 Martin Matusiak numerodix@gmail.com 9f374865ab0d5c2ae1318d111706e80033505c4a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 2008-07-07T04:03:45-07:00 2008-07-07T04:03:45-07:00 fix regression in ftp resume 602383b7bae34ba0750a01e1506054127c85f1f8 Martin Matusiak numerodix@gmail.com b260b6a78f1fd171b6120cd2dab6466680bfa741 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/9f374865ab0d5c2ae1318d111706e80033505c4a 9f374865ab0d5c2ae1318d111706e80033505c4a 2008-07-06T03:42:38-07:00 2008-07-06T03:42:38-07:00 quote pattern in opts help a0a62084c2b074a1a2a3e6f056053043e848041b Martin Matusiak numerodix@gmail.com