15022307449ba681507e5dfa0c49b3feed7d5ea6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/07e951cee00a15ebc85b8dd4efa6841da494d9ee 07e951cee00a15ebc85b8dd4efa6841da494d9ee 2008-07-20T16:37:48-07:00 2008-07-19T15:06:31-07:00 . 7278fad9f6955b678149ef463022b08533aac73e Martin Matusiak numerodix@gmail.com 62dd9e94779ca8fb259c69f29bdc737b156f3e16 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/15022307449ba681507e5dfa0c49b3feed7d5ea6 15022307449ba681507e5dfa0c49b3feed7d5ea6 2008-07-20T16:37:06-07:00 2008-07-19T14:09:00-07:00 messy 574541cf24ae075fbf74137ef13bbb20e2a96d0a Martin Matusiak numerodix@gmail.com d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/62dd9e94779ca8fb259c69f29bdc737b156f3e16 62dd9e94779ca8fb259c69f29bdc737b156f3e16 2008-07-19T11:15:42-07:00 2008-07-19T11:15:42-07:00 make queue and wb instance variables 084e09bf07edf9f28098f2b2cb6761f7caf0d567 Martin Matusiak numerodix@gmail.com 554af5a465a020adf72903f094bd96987449fe10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 2008-07-19T09:08:58-07:00 2008-07-19T09:08:58-07:00 wrap spiderfetch methods in class, manage state variables better bb3b948e4308b08593c22a951e896962188458c1 Martin Matusiak numerodix@gmail.com 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/554af5a465a020adf72903f094bd96987449fe10 554af5a465a020adf72903f094bd96987449fe10 2008-07-17T14:48:23-07:00 2008-07-17T14:48:23-07:00 spider.py also supports local files, include that in the help d164fbf3356e354a72da509b1714d55b36373dce Martin Matusiak numerodix@gmail.com 5d2f989aee8ea812b52c812aef5149f69c036a10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 2008-07-17T11:37:59-07:00 2008-07-17T11:37:59-07:00 periodically save state 6f4782d744c4e5cc88be907d0857857a0e483923 Martin Matusiak numerodix@gmail.com 67359833ad7a2d8e2b71794902cc10ee77b43845 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5d2f989aee8ea812b52c812aef5149f69c036a10 5d2f989aee8ea812b52c812aef5149f69c036a10 2008-07-17T10:56:58-07:00 2008-07-17T10:56:58-07:00 safer serialization by writing to .partial files 97f8d3e84d92035e6d8a1779fbf2ba4019aa4c0c Martin Matusiak numerodix@gmail.com 73432359c7508fb7b74ef08e16f0457ea81536d2 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/67359833ad7a2d8e2b71794902cc10ee77b43845 67359833ad7a2d8e2b71794902cc10ee77b43845 2008-07-09T14:57:56-07:00 2008-07-09T14:57:56-07:00 formatting 3181f7bf905381eeaffbd76a0878fc900344883a Martin Matusiak numerodix@gmail.com 1ff1a04b833f4e01fa530612c80ae73cde49d9a6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/73432359c7508fb7b74ef08e16f0457ea81536d2 73432359c7508fb7b74ef08e16f0457ea81536d2 2008-07-08T18:37:32-07:00 2008-07-08T18:37:32-07:00 avoid division by zero 1b35ae6c0328a873ad37656e587c930f537de4c0 Martin Matusiak numerodix@gmail.com c6515b4b460cd2805b8086b1045e48651d643814 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1ff1a04b833f4e01fa530612c80ae73cde49d9a6 1ff1a04b833f4e01fa530612c80ae73cde49d9a6 2008-07-08T13:13:16-07:00 2008-07-08T13:13:16-07:00 fix naming bug related to -c ed03a81a9540e547ee3f8e56f7550013ecec0b76 Martin Matusiak numerodix@gmail.com d4f6f84c8ffb3dfe8076554afe3582856798a7a3 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c6515b4b460cd2805b8086b1045e48651d643814 c6515b4b460cd2805b8086b1045e48651d643814 2008-07-08T12:52:37-07:00 2008-07-08T12:52:37-07:00 add -t option to fetcher 41c4dca76eadcbac345723d8be55206a96617519 Martin Matusiak numerodix@gmail.com 1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d4f6f84c8ffb3dfe8076554afe3582856798a7a3 d4f6f84c8ffb3dfe8076554afe3582856798a7a3 2008-07-08T10:33:40-07:00 2008-07-08T10:33:40-07:00 file naming fix 21936e7dcc8add399c900fc7711caab2051e6b44 Martin Matusiak numerodix@gmail.com 9f374865ab0d5c2ae1318d111706e80033505c4a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 1ffaddb1fcdd04fbf39156ec2114aefe3da34a92 2008-07-07T04:03:45-07:00 2008-07-07T04:03:45-07:00 fix regression in ftp resume 602383b7bae34ba0750a01e1506054127c85f1f8 Martin Matusiak numerodix@gmail.com b260b6a78f1fd171b6120cd2dab6466680bfa741 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/9f374865ab0d5c2ae1318d111706e80033505c4a 9f374865ab0d5c2ae1318d111706e80033505c4a 2008-07-06T03:42:38-07:00 2008-07-06T03:42:38-07:00 quote pattern in opts help a0a62084c2b074a1a2a3e6f056053043e848041b Martin Matusiak numerodix@gmail.com a4e3f0180fd8c719dc732b96aa7c5b95496a6fc1 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b260b6a78f1fd171b6120cd2dab6466680bfa741 b260b6a78f1fd171b6120cd2dab6466680bfa741 2008-07-02T12:32:52-07:00 2008-07-02T12:32:52-07:00 minor 3f1756954adfdaeaeda205973f6c6c8fd92ff7b2 Martin Matusiak numerodix@gmail.com 829733bd001d5c40f7e5d7b9c59a78a31c9e8b18 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/a4e3f0180fd8c719dc732b96aa7c5b95496a6fc1 a4e3f0180fd8c719dc732b96aa7c5b95496a6fc1 2008-07-02T11:21:28-07:00 2008-07-02T11:21:28-07:00 add testcases f54ae99d890d1e6610dc4a1d987ebb64395d3c0e Martin Matusiak numerodix@gmail.com 5d6d356ea6f83b3934552d57fcae3daf8624634f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/829733bd001d5c40f7e5d7b9c59a78a31c9e8b18 829733bd001d5c40f7e5d7b9c59a78a31c9e8b18 2008-07-02T05:08:04-07:00 2008-07-02T05:08:04-07:00 add regexs for matching unquoted urls d8c28620dea509046faa1f86ee9ca1928dbaf40a Martin Matusiak numerodix@gmail.com 522fdd03861da50dead30a604dcec3c2d646f058 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5d6d356ea6f83b3934552d57fcae3daf8624634f 5d6d356ea6f83b3934552d57fcae3daf8624634f 2008-07-01T14:11:00-07:00 2008-07-01T14:11:00-07:00 fix silly filename inconsistency 9c1faf01a066609c3d418fa5fa35b0a66315e3b1 Martin Matusiak numerodix@gmail.com ab48bc600f92b6d15159563b3f579d527b4a7bc5 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/522fdd03861da50dead30a604dcec3c2d646f058 522fdd03861da50dead30a604dcec3c2d646f058 2008-07-01T08:32:22-07:00 2008-07-01T08:32:22-07:00 don't try to resume zero size files 6da577288fb9a7ff30bbdf17239b9353d49cefbf Martin Matusiak numerodix@gmail.com 762d679340a3244e34faf24a83700059271a1c0a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/ab48bc600f92b6d15159563b3f579d527b4a7bc5 ab48bc600f92b6d15159563b3f579d527b4a7bc5 2008-07-01T06:39:46-07:00 2008-07-01T06:39:46-07:00 handle lack of http resume support 062b76de979d1319984ccfcafe577e8c781c871d Martin Matusiak numerodix@gmail.com 4c6187410573d30f0c38d2b6bd6c2bd1d97c4743 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/762d679340a3244e34faf24a83700059271a1c0a 762d679340a3244e34faf24a83700059271a1c0a 2008-06-30T14:33:38-07:00 2008-06-30T14:33:38-07:00 add switch for long filenames ad8c5c31da526730c2f4b61164cefcb8633378e5 Martin Matusiak numerodix@gmail.com 2e7260239892a7453f1148c3da8a73663daffd7a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/4c6187410573d30f0c38d2b6bd6c2bd1d97c4743 4c6187410573d30f0c38d2b6bd6c2bd1d97c4743 2008-06-29T14:43:36-07:00 2008-06-29T14:43:36-07:00 bugfix 65b7b7f20d999a202b5fa6eae22b284dd5d12153 Martin Matusiak numerodix@gmail.com 33d69373f5e6736f7abc115d12c3a214a44f898e Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2e7260239892a7453f1148c3da8a73663daffd7a 2e7260239892a7453f1148c3da8a73663daffd7a 2008-06-29T13:31:38-07:00 2008-06-29T13:31:38-07:00 fetch: add support for multiple urls on cmd line 333ebee39b493522c9295d326f59861bd9ba8a98 Martin Matusiak numerodix@gmail.com 8d24afdca0621fcc512b2733f1af3646b59027e7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/33d69373f5e6736f7abc115d12c3a214a44f898e 33d69373f5e6736f7abc115d12c3a214a44f898e 2008-06-29T13:18:29-07:00 2008-06-29T13:18:29-07:00 minor 655d385cc5d0a07738886f1af631d4480233fdab Martin Matusiak numerodix@gmail.com b84e37a4c1de6a970d3bd0ed0194c6a725192148 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/8d24afdca0621fcc512b2733f1af3646b59027e7 8d24afdca0621fcc512b2733f1af3646b59027e7 2008-06-29T13:12:43-07:00 2008-06-29T13:12:43-07:00 minor d53c3b256d836402c5bcad68b3a568dfa378b517 Martin Matusiak numerodix@gmail.com b42d8c67351d1868c5b75316ba62bcc02319266d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b84e37a4c1de6a970d3bd0ed0194c6a725192148 b84e37a4c1de6a970d3bd0ed0194c6a725192148 2008-06-29T13:02:37-07:00 2008-06-29T13:02:37-07:00 add code for resuming transfers, needs a lot of cleaning up 5736c1da08c77f7fd2cdc6d29c1b0a65595b80d7 Martin Matusiak numerodix@gmail.com 03b5384ed9553261c7551731ada4b549c34ded9b Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b42d8c67351d1868c5b75316ba62bcc02319266d b42d8c67351d1868c5b75316ba62bcc02319266d 2008-06-28T16:08:55-07:00 2008-06-28T16:08:55-07:00 move temporal fetch error to its class c3c1c5048d548afaee46e67a8453cbb15c23f785 Martin Matusiak numerodix@gmail.com 10dea686f51fdd7c0555f1a4280349f82b9e97d7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/03b5384ed9553261c7551731ada4b549c34ded9b 03b5384ed9553261c7551731ada4b549c34ded9b 2008-06-28T14:15:07-07:00 2008-06-28T14:15:07-07:00 add logic for retrying fetch on certain errors bd6caebde9f99cff3bab28e14cbd8b6d30a92d56 Martin Matusiak numerodix@gmail.com 528ebc524ba095442f014274b44f837bf6fd346a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/10dea686f51fdd7c0555f1a4280349f82b9e97d7 10dea686f51fdd7c0555f1a4280349f82b9e97d7 2008-06-28T12:44:39-07:00 2008-06-28T12:44:39-07:00 switch fetch errors from string based to object based 4edddc3f34e3c318694af017e88df1cf0c8aebe1 Martin Matusiak numerodix@gmail.com 88c485021d81f5e1f9f50e3b0afcebb9752c1533 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/528ebc524ba095442f014274b44f837bf6fd346a 528ebc524ba095442f014274b44f837bf6fd346a 2008-06-28T09:27:39-07:00 2008-06-28T09:27:39-07:00 fix length bug in truncate_url and preformat url truncation fc6740e979fce6f75c19f7a893b06ae6d3f20090 Martin Matusiak numerodix@gmail.com