33d69373f5e6736f7abc115d12c3a214a44f898e Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2e7260239892a7453f1148c3da8a73663daffd7a 2e7260239892a7453f1148c3da8a73663daffd7a 2008-06-29T13:31:38-07:00 2008-06-29T13:31:38-07:00 fetch: add support for multiple urls on cmd line 333ebee39b493522c9295d326f59861bd9ba8a98 Martin Matusiak numerodix@gmail.com 8d24afdca0621fcc512b2733f1af3646b59027e7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/33d69373f5e6736f7abc115d12c3a214a44f898e 33d69373f5e6736f7abc115d12c3a214a44f898e 2008-06-29T13:18:29-07:00 2008-06-29T13:18:29-07:00 minor 655d385cc5d0a07738886f1af631d4480233fdab Martin Matusiak numerodix@gmail.com b84e37a4c1de6a970d3bd0ed0194c6a725192148 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/8d24afdca0621fcc512b2733f1af3646b59027e7 8d24afdca0621fcc512b2733f1af3646b59027e7 2008-06-29T13:12:43-07:00 2008-06-29T13:12:43-07:00 minor d53c3b256d836402c5bcad68b3a568dfa378b517 Martin Matusiak numerodix@gmail.com b42d8c67351d1868c5b75316ba62bcc02319266d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b84e37a4c1de6a970d3bd0ed0194c6a725192148 b84e37a4c1de6a970d3bd0ed0194c6a725192148 2008-06-29T13:02:37-07:00 2008-06-29T13:02:37-07:00 add code for resuming transfers, needs a lot of cleaning up 5736c1da08c77f7fd2cdc6d29c1b0a65595b80d7 Martin Matusiak numerodix@gmail.com 03b5384ed9553261c7551731ada4b549c34ded9b Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b42d8c67351d1868c5b75316ba62bcc02319266d b42d8c67351d1868c5b75316ba62bcc02319266d 2008-06-28T16:08:55-07:00 2008-06-28T16:08:55-07:00 move temporal fetch error to its class c3c1c5048d548afaee46e67a8453cbb15c23f785 Martin Matusiak numerodix@gmail.com 10dea686f51fdd7c0555f1a4280349f82b9e97d7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/03b5384ed9553261c7551731ada4b549c34ded9b 03b5384ed9553261c7551731ada4b549c34ded9b 2008-06-28T14:15:07-07:00 2008-06-28T14:15:07-07:00 add logic for retrying fetch on certain errors bd6caebde9f99cff3bab28e14cbd8b6d30a92d56 Martin Matusiak numerodix@gmail.com 528ebc524ba095442f014274b44f837bf6fd346a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/10dea686f51fdd7c0555f1a4280349f82b9e97d7 10dea686f51fdd7c0555f1a4280349f82b9e97d7 2008-06-28T12:44:39-07:00 2008-06-28T12:44:39-07:00 switch fetch errors from string based to object based 4edddc3f34e3c318694af017e88df1cf0c8aebe1 Martin Matusiak numerodix@gmail.com 88c485021d81f5e1f9f50e3b0afcebb9752c1533 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/528ebc524ba095442f014274b44f837bf6fd346a 528ebc524ba095442f014274b44f837bf6fd346a 2008-06-28T09:27:39-07:00 2008-06-28T09:27:39-07:00 fix length bug in truncate_url and preformat url truncation fc6740e979fce6f75c19f7a893b06ae6d3f20090 Martin Matusiak numerodix@gmail.com 58f98bf8cf3b803880875253a94d266cf959a058 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/88c485021d81f5e1f9f50e3b0afcebb9752c1533 88c485021d81f5e1f9f50e3b0afcebb9752c1533 2008-06-28T06:10:02-07:00 2008-06-28T06:10:02-07:00 set http referer in fetch.py af0480d8c123aa893918119870b29e911b698ced Martin Matusiak numerodix@gmail.com af12380761fee007e6e1923923f421afef7cee85 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/58f98bf8cf3b803880875253a94d266cf959a058 58f98bf8cf3b803880875253a94d266cf959a058 2008-06-28T05:46:12-07:00 2008-06-28T05:46:12-07:00 fix http redirect logic when fetch.py == sys.argv[0] 328605c47aa96351a7a69043db6756ab63668ae0 Martin Matusiak numerodix@gmail.com 4f070b40fffe53eae992a23e9811281b8dea55bb Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/af12380761fee007e6e1923923f421afef7cee85 af12380761fee007e6e1923923f421afef7cee85 2008-06-27T18:35:38-07:00 2008-06-27T18:35:38-07:00 add option to use standard opener in Fetcher b421dd3e4714eb63cf3430e7cfb6f73673b7ce75 Martin Matusiak numerodix@gmail.com 4059f838813c529a712f7dfee5f295b053e20d01 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/4f070b40fffe53eae992a23e9811281b8dea55bb 4f070b40fffe53eae992a23e9811281b8dea55bb 2008-06-27T17:34:39-07:00 2008-06-27T17:34:39-07:00 fix spidering colorize 3040c6e5affcf7507b74b763153ea4cb1f511971 Martin Matusiak numerodix@gmail.com c5130c4195cddfdd52cdf79ef4df7186e443d8c7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/4059f838813c529a712f7dfee5f295b053e20d01 4059f838813c529a712f7dfee5f295b053e20d01 2008-06-27T14:48:09-07:00 2008-06-27T14:48:09-07:00 output formatting f72d05bb2325623a459d67c0f0ea333b7ffbadae Martin Matusiak numerodix@gmail.com 8f6c5d066b1eea6d894f99509aa2072b2efeab90 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c5130c4195cddfdd52cdf79ef4df7186e443d8c7 c5130c4195cddfdd52cdf79ef4df7186e443d8c7 2008-06-27T14:30:21-07:00 2008-06-27T14:30:21-07:00 add help for env vars a7946633ede895f6319cacfcf6ae728493114fd9 Martin Matusiak numerodix@gmail.com e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/8f6c5d066b1eea6d894f99509aa2072b2efeab90 8f6c5d066b1eea6d894f99509aa2072b2efeab90 2008-06-27T14:12:29-07:00 2008-06-27T14:12:29-07:00 add descriptions of the tools e6c31458faa9d382324c9b3081a6c9b29967c17d Martin Matusiak numerodix@gmail.com 78874ff665d7f6c4403b039b0af6bc0a2bffcb19 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 2008-06-27T13:33:20-07:00 2008-06-27T13:33:20-07:00 rewrite opts logic 46fb041a9ba955bf1f8396698d5057ae94e9f44b Martin Matusiak numerodix@gmail.com 3556b280d07ca9b019b5c41ca6690998c6dd5a07 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/78874ff665d7f6c4403b039b0af6bc0a2bffcb19 78874ff665d7f6c4403b039b0af6bc0a2bffcb19 2008-06-27T12:53:00-07:00 2008-06-27T12:53:00-07:00 minor 896faefe00fb2205d738647fd3f6b498849e618f Martin Matusiak numerodix@gmail.com 5fcb044cf5b4c43bde858e59d3acc6de87b8092d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/3556b280d07ca9b019b5c41ca6690998c6dd5a07 3556b280d07ca9b019b5c41ca6690998c6dd5a07 2008-06-27T11:56:29-07:00 2008-06-27T11:56:29-07:00 add pattern for matching ftp dir listings 111577e764718537c1801ef4fa09f8afd1827bb2 Martin Matusiak numerodix@gmail.com eda049dfd6c600c99c1de34ec0240139dde742b1 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5fcb044cf5b4c43bde858e59d3acc6de87b8092d 5fcb044cf5b4c43bde858e59d3acc6de87b8092d 2008-06-27T10:11:36-07:00 2008-06-27T10:11:36-07:00 use download file size instead of checking urllib headers 4b9cf3452303dc5988b1fafd30ce369ad7398068 Martin Matusiak numerodix@gmail.com 49f20e35e9b289d39cb8ea934db6de11f371e95f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/eda049dfd6c600c99c1de34ec0240139dde742b1 eda049dfd6c600c99c1de34ec0240139dde742b1 2008-06-27T10:10:49-07:00 2008-06-27T10:10:49-07:00 kill faulty logdir() wrapping 19866618abc4957393c53f62007a417808e9f3cf Martin Matusiak numerodix@gmail.com 87ad618134a043cb6400bdddcdb381664df9ba32 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/49f20e35e9b289d39cb8ea934db6de11f371e95f 49f20e35e9b289d39cb8ea934db6de11f371e95f 2008-06-27T09:03:23-07:00 2008-06-27T09:03:23-07:00 fix bugs that caused fetching to halt after two redirects make redirects explicit to be more clear on what is happening efb98581205c636f656039d9043099889ce66db0 Martin Matusiak numerodix@gmail.com d8ec49d54707aff5119dc62a30413b513118d34c Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/87ad618134a043cb6400bdddcdb381664df9ba32 87ad618134a043cb6400bdddcdb381664df9ba32 2008-06-27T07:30:27-07:00 2008-06-27T07:30:27-07:00 SOCKET_TIMETOUT is env var cabc5b3c2b82e74207535263cc7e11c994f0ac06 Martin Matusiak numerodix@gmail.com e44f12958c4c4e5b31b7c2a596767879e5f106ec Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d8ec49d54707aff5119dc62a30413b513118d34c d8ec49d54707aff5119dc62a30413b513118d34c 2008-06-27T07:28:26-07:00 2008-06-27T07:28:26-07:00 fix io.py sanity check, apply logdir() only on paths == basename 0b0f47d4f68ba9540f6c060d140dfe504096f317 Martin Matusiak numerodix@gmail.com e5068d31e916afb58b044644040f1983dc3bb076 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e44f12958c4c4e5b31b7c2a596767879e5f106ec e44f12958c4c4e5b31b7c2a596767879e5f106ec 2008-06-27T06:41:40-07:00 2008-06-27T06:41:40-07:00 fix for short filenames that are null 907aaea97ed263082182d27bec1dfd2cd46ed0a8 Martin Matusiak numerodix@gmail.com 17c7c8b61c88d8502d7aebcee2af0c69912dff96 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e5068d31e916afb58b044644040f1983dc3bb076 e5068d31e916afb58b044644040f1983dc3bb076 2008-06-27T06:38:23-07:00 2008-06-27T06:38:23-07:00 require $TERM set to output color abafd7437d8dd3a32bab14d54aaad4fe54c2ed9e Martin Matusiak numerodix@gmail.com e9464d66550733b307c1c75a1c69264d1f45eef4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/17c7c8b61c88d8502d7aebcee2af0c69912dff96 17c7c8b61c88d8502d7aebcee2af0c69912dff96 2008-06-27T05:36:33-07:00 2008-06-27T05:36:33-07:00 disable gzip, it's slow :( 3067b7a29963fd417f84e52cce8c621c81f2fd79 Martin Matusiak numerodix@gmail.com 3e9bb240e2bcbf12ad1798646387e10d2c9a7bce Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e9464d66550733b307c1c75a1c69264d1f45eef4 e9464d66550733b307c1c75a1c69264d1f45eef4 2008-06-27T05:22:38-07:00 2008-06-27T05:22:38-07:00 small fixes to fetch.py df016db6243610cc625c28e8e059575858fbd6d7 Martin Matusiak numerodix@gmail.com f6e0031945fd9882ac14daa0a483d70f0b2c1bcc Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/3e9bb240e2bcbf12ad1798646387e10d2c9a7bce 3e9bb240e2bcbf12ad1798646387e10d2c9a7bce 2008-06-27T05:07:05-07:00 2008-06-27T05:07:05-07:00 extract queue splitting logic to split_queue. better, but still not pretty 2891632aa45ccb88a5940f7b5572c2ffe21b7a06 Martin Matusiak numerodix@gmail.com 16044c5b6927730ea8bafabe6b5a5fcd73e033ca Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/f6e0031945fd9882ac14daa0a483d70f0b2c1bcc f6e0031945fd9882ac14daa0a483d70f0b2c1bcc 2008-06-27T04:10:09-07:00 2008-06-27T04:10:09-07:00 don't use 'tuple' as identifier 0e8c72c9c3218031e0f7e95ca3548bb9c0931c18 Martin Matusiak numerodix@gmail.com 17e168841cad2cd9015c06854ac6024c1ca86ad7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/16044c5b6927730ea8bafabe6b5a5fcd73e033ca 16044c5b6927730ea8bafabe6b5a5fcd73e033ca 2008-06-26T19:14:34-07:00 2008-06-26T19:14:34-07:00 minor d69fed0431e4a538684e7b7c9165417d5ca74163 Martin Matusiak numerodix@gmail.com