c5130c4195cddfdd52cdf79ef4df7186e443d8c7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/4059f838813c529a712f7dfee5f295b053e20d01 4059f838813c529a712f7dfee5f295b053e20d01 2008-06-27T14:48:09-07:00 2008-06-27T14:48:09-07:00 output formatting f72d05bb2325623a459d67c0f0ea333b7ffbadae Martin Matusiak numerodix@gmail.com 8f6c5d066b1eea6d894f99509aa2072b2efeab90 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c5130c4195cddfdd52cdf79ef4df7186e443d8c7 c5130c4195cddfdd52cdf79ef4df7186e443d8c7 2008-06-27T14:30:21-07:00 2008-06-27T14:30:21-07:00 add help for env vars a7946633ede895f6319cacfcf6ae728493114fd9 Martin Matusiak numerodix@gmail.com e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/8f6c5d066b1eea6d894f99509aa2072b2efeab90 8f6c5d066b1eea6d894f99509aa2072b2efeab90 2008-06-27T14:12:29-07:00 2008-06-27T14:12:29-07:00 add descriptions of the tools e6c31458faa9d382324c9b3081a6c9b29967c17d Martin Matusiak numerodix@gmail.com 78874ff665d7f6c4403b039b0af6bc0a2bffcb19 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 2008-06-27T13:33:20-07:00 2008-06-27T13:33:20-07:00 rewrite opts logic 46fb041a9ba955bf1f8396698d5057ae94e9f44b Martin Matusiak numerodix@gmail.com 3556b280d07ca9b019b5c41ca6690998c6dd5a07 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/78874ff665d7f6c4403b039b0af6bc0a2bffcb19 78874ff665d7f6c4403b039b0af6bc0a2bffcb19 2008-06-27T12:53:00-07:00 2008-06-27T12:53:00-07:00 minor 896faefe00fb2205d738647fd3f6b498849e618f Martin Matusiak numerodix@gmail.com 5fcb044cf5b4c43bde858e59d3acc6de87b8092d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/3556b280d07ca9b019b5c41ca6690998c6dd5a07 3556b280d07ca9b019b5c41ca6690998c6dd5a07 2008-06-27T11:56:29-07:00 2008-06-27T11:56:29-07:00 add pattern for matching ftp dir listings 111577e764718537c1801ef4fa09f8afd1827bb2 Martin Matusiak numerodix@gmail.com eda049dfd6c600c99c1de34ec0240139dde742b1 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5fcb044cf5b4c43bde858e59d3acc6de87b8092d 5fcb044cf5b4c43bde858e59d3acc6de87b8092d 2008-06-27T10:11:36-07:00 2008-06-27T10:11:36-07:00 use download file size instead of checking urllib headers 4b9cf3452303dc5988b1fafd30ce369ad7398068 Martin Matusiak numerodix@gmail.com 49f20e35e9b289d39cb8ea934db6de11f371e95f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/eda049dfd6c600c99c1de34ec0240139dde742b1 eda049dfd6c600c99c1de34ec0240139dde742b1 2008-06-27T10:10:49-07:00 2008-06-27T10:10:49-07:00 kill faulty logdir() wrapping 19866618abc4957393c53f62007a417808e9f3cf Martin Matusiak numerodix@gmail.com 87ad618134a043cb6400bdddcdb381664df9ba32 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/49f20e35e9b289d39cb8ea934db6de11f371e95f 49f20e35e9b289d39cb8ea934db6de11f371e95f 2008-06-27T09:03:23-07:00 2008-06-27T09:03:23-07:00 fix bugs that caused fetching to halt after two redirects make redirects explicit to be more clear on what is happening efb98581205c636f656039d9043099889ce66db0 Martin Matusiak numerodix@gmail.com d8ec49d54707aff5119dc62a30413b513118d34c Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/87ad618134a043cb6400bdddcdb381664df9ba32 87ad618134a043cb6400bdddcdb381664df9ba32 2008-06-27T07:30:27-07:00 2008-06-27T07:30:27-07:00 SOCKET_TIMETOUT is env var cabc5b3c2b82e74207535263cc7e11c994f0ac06 Martin Matusiak numerodix@gmail.com e44f12958c4c4e5b31b7c2a596767879e5f106ec Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d8ec49d54707aff5119dc62a30413b513118d34c d8ec49d54707aff5119dc62a30413b513118d34c 2008-06-27T07:28:26-07:00 2008-06-27T07:28:26-07:00 fix io.py sanity check, apply logdir() only on paths == basename 0b0f47d4f68ba9540f6c060d140dfe504096f317 Martin Matusiak numerodix@gmail.com e5068d31e916afb58b044644040f1983dc3bb076 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e44f12958c4c4e5b31b7c2a596767879e5f106ec e44f12958c4c4e5b31b7c2a596767879e5f106ec 2008-06-27T06:41:40-07:00 2008-06-27T06:41:40-07:00 fix for short filenames that are null 907aaea97ed263082182d27bec1dfd2cd46ed0a8 Martin Matusiak numerodix@gmail.com 17c7c8b61c88d8502d7aebcee2af0c69912dff96 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e5068d31e916afb58b044644040f1983dc3bb076 e5068d31e916afb58b044644040f1983dc3bb076 2008-06-27T06:38:23-07:00 2008-06-27T06:38:23-07:00 require $TERM set to output color abafd7437d8dd3a32bab14d54aaad4fe54c2ed9e Martin Matusiak numerodix@gmail.com e9464d66550733b307c1c75a1c69264d1f45eef4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/17c7c8b61c88d8502d7aebcee2af0c69912dff96 17c7c8b61c88d8502d7aebcee2af0c69912dff96 2008-06-27T05:36:33-07:00 2008-06-27T05:36:33-07:00 disable gzip, it's slow :( 3067b7a29963fd417f84e52cce8c621c81f2fd79 Martin Matusiak numerodix@gmail.com 3e9bb240e2bcbf12ad1798646387e10d2c9a7bce Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e9464d66550733b307c1c75a1c69264d1f45eef4 e9464d66550733b307c1c75a1c69264d1f45eef4 2008-06-27T05:22:38-07:00 2008-06-27T05:22:38-07:00 small fixes to fetch.py df016db6243610cc625c28e8e059575858fbd6d7 Martin Matusiak numerodix@gmail.com f6e0031945fd9882ac14daa0a483d70f0b2c1bcc Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/3e9bb240e2bcbf12ad1798646387e10d2c9a7bce 3e9bb240e2bcbf12ad1798646387e10d2c9a7bce 2008-06-27T05:07:05-07:00 2008-06-27T05:07:05-07:00 extract queue splitting logic to split_queue. better, but still not pretty 2891632aa45ccb88a5940f7b5572c2ffe21b7a06 Martin Matusiak numerodix@gmail.com 16044c5b6927730ea8bafabe6b5a5fcd73e033ca Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/f6e0031945fd9882ac14daa0a483d70f0b2c1bcc f6e0031945fd9882ac14daa0a483d70f0b2c1bcc 2008-06-27T04:10:09-07:00 2008-06-27T04:10:09-07:00 don't use 'tuple' as identifier 0e8c72c9c3218031e0f7e95ca3548bb9c0931c18 Martin Matusiak numerodix@gmail.com 17e168841cad2cd9015c06854ac6024c1ca86ad7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/16044c5b6927730ea8bafabe6b5a5fcd73e033ca 16044c5b6927730ea8bafabe6b5a5fcd73e033ca 2008-06-26T19:14:34-07:00 2008-06-26T19:14:34-07:00 minor d69fed0431e4a538684e7b7c9165417d5ca74163 Martin Matusiak numerodix@gmail.com f0b16f45aa11013fdc88d4be2136a5e99c4069f9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/17e168841cad2cd9015c06854ac6024c1ca86ad7 17e168841cad2cd9015c06854ac6024c1ca86ad7 2008-06-26T19:11:10-07:00 2008-06-26T19:11:10-07:00 minor bfc640c73ce840441154047b822cbd3e9a214d65 Martin Matusiak numerodix@gmail.com 63ebf396c0ec7006095d3c5cf554ed8beae83e21 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/f0b16f45aa11013fdc88d4be2136a5e99c4069f9 f0b16f45aa11013fdc88d4be2136a5e99c4069f9 2008-06-26T18:52:24-07:00 2008-06-26T18:52:24-07:00 modularize and cleanup f910bcbd324b1c4defbfd251009ae746d7ae5fe5 Martin Matusiak numerodix@gmail.com 5c68bd93977ec11a38a06892ea398f373492c2fe Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/63ebf396c0ec7006095d3c5cf554ed8beae83e21 63ebf396c0ec7006095d3c5cf554ed8beae83e21 2008-06-26T16:33:32-07:00 2008-06-26T16:33:32-07:00 extract url classifying logic to function d88a48e0dd2d0c58503f5e91cafe94b4541eabf3 Martin Matusiak numerodix@gmail.com a1b75eacf9703c6597ac69fe3143ea70c0a0801f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5c68bd93977ec11a38a06892ea398f373492c2fe 5c68bd93977ec11a38a06892ea398f373492c2fe 2008-06-26T15:35:22-07:00 2008-06-26T15:35:22-07:00 fix 503 logic b7816e061af4da5d25ae35e7e02603e850124de3 Martin Matusiak numerodix@gmail.com 60f97aaec9e811ae95433e603b622910282b9971 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/a1b75eacf9703c6597ac69fe3143ea70c0a0801f a1b75eacf9703c6597ac69fe3143ea70c0a0801f 2008-06-26T14:44:10-07:00 2008-06-26T14:44:10-07:00 add logic for retrying on 503 69017b4bccc3e52471f0ea63204d1d96613e3edb Martin Matusiak numerodix@gmail.com d867a541a4326113e6cecfeb2e75a46f986207ac Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/60f97aaec9e811ae95433e603b622910282b9971 60f97aaec9e811ae95433e603b622910282b9971 2008-06-26T12:39:35-07:00 2008-06-26T12:39:35-07:00 rewrite urls on redirect also 8894eec0252477ee01fb723421e8008895afaa81 Martin Matusiak numerodix@gmail.com ed6cb7eeacd43b731cc2aeda1ceaf421d6f9960f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d867a541a4326113e6cecfeb2e75a46f986207ac d867a541a4326113e6cecfeb2e75a46f986207ac 2008-06-26T10:32:05-07:00 2008-06-26T10:32:05-07:00 docs 18440cacea44d9308d24b5821a396ec062dc18fd Martin Matusiak numerodix@gmail.com 08d09ae37448cf6e97a50ac2c4834ff4dec9e7f9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/ed6cb7eeacd43b731cc2aeda1ceaf421d6f9960f ed6cb7eeacd43b731cc2aeda1ceaf421d6f9960f 2008-06-26T10:01:16-07:00 2008-06-26T10:01:16-07:00 minor 75d472c4e08f13ecea36b7532d687fa33384ad23 Martin Matusiak numerodix@gmail.com 151ade13aba034901dd6202aa332b5ec4fa39d0a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/08d09ae37448cf6e97a50ac2c4834ff4dec9e7f9 08d09ae37448cf6e97a50ac2c4834ff4dec9e7f9 2008-06-26T09:55:33-07:00 2008-06-26T09:55:33-07:00 chmod +x c1b6e777c4ce88d0eef70da886b2943f0c2418d4 Martin Matusiak numerodix@gmail.com dee0d2edc41de144eb89622091445e11c1beca8f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/151ade13aba034901dd6202aa332b5ec4fa39d0a 151ade13aba034901dd6202aa332b5ec4fa39d0a 2008-06-26T09:50:20-07:00 2008-06-26T09:50:20-07:00 remove cruft 756c52fbcada1fea03b2da07c7a8a16995831f63 Martin Matusiak numerodix@gmail.com 7e2d2c817c936c8df00ab69e0445525fde3bcaa3 d7eecc2ded2105de8cf971abc5e53309eea23fbd Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/dee0d2edc41de144eb89622091445e11c1beca8f dee0d2edc41de144eb89622091445e11c1beca8f 2008-06-26T09:38:10-07:00 2008-06-26T09:37:33-07:00 Merge spiderfetch2 into python-rewrite 1f257f7e77e2615e3629241d0c9b245f61bc0f0f Martin Matusiak numerodix@gmail.com 4e440d3b47797627b2c7268990a3367bec95c247 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d7eecc2ded2105de8cf971abc5e53309eea23fbd d7eecc2ded2105de8cf971abc5e53309eea23fbd 2008-06-26T08:35:52-07:00 2008-06-26T08:35:52-07:00 change videostream recipe to find urls at any depth 756c52fbcada1fea03b2da07c7a8a16995831f63 Martin Matusiak numerodix@gmail.com