numerodix / spiderfetch fork watch download tarball
public
Forks0Right
Watchers5Right
Description: A modular web spider
Homepage: http://www.matusiak.eu/numerodix/code.php
Clone URL: git://github.com/numerodix/spiderfetch.git
spiderfetch / Commit History feed

2008-06-27

output formatting
numerodix (author)
Fri Jun 27 14:48:09 -0700 2008
commit  4059f838813c529a712f7dfee5f295b053e20d01
tree    f72d05bb2325623a459d67c0f0ea333b7ffbadae
parent  c5130c4195cddfdd52cdf79ef4df7186e443d8c7 
add help for env vars
numerodix (author)
Fri Jun 27 14:30:21 -0700 2008
commit  c5130c4195cddfdd52cdf79ef4df7186e443d8c7
tree    a7946633ede895f6319cacfcf6ae728493114fd9
parent  8f6c5d066b1eea6d894f99509aa2072b2efeab90 
add descriptions of the tools
numerodix (author)
Fri Jun 27 14:12:29 -0700 2008
commit  8f6c5d066b1eea6d894f99509aa2072b2efeab90
tree    e6c31458faa9d382324c9b3081a6c9b29967c17d
parent  e93e1634c9f75343401e9ab7a9fe20aa78f24ac3 
rewrite opts logic
numerodix (author)
Fri Jun 27 13:33:20 -0700 2008
commit  e93e1634c9f75343401e9ab7a9fe20aa78f24ac3
tree    46fb041a9ba955bf1f8396698d5057ae94e9f44b
parent  78874ff665d7f6c4403b039b0af6bc0a2bffcb19 
minor
numerodix (author)
Fri Jun 27 12:53:00 -0700 2008
commit  78874ff665d7f6c4403b039b0af6bc0a2bffcb19
tree    896faefe00fb2205d738647fd3f6b498849e618f
parent  3556b280d07ca9b019b5c41ca6690998c6dd5a07 
add pattern for matching ftp dir listings
numerodix (author)
Fri Jun 27 11:56:29 -0700 2008
commit  3556b280d07ca9b019b5c41ca6690998c6dd5a07
tree    111577e764718537c1801ef4fa09f8afd1827bb2
parent  5fcb044cf5b4c43bde858e59d3acc6de87b8092d 
use download file size instead of checking urllib headers
numerodix (author)
Fri Jun 27 10:11:36 -0700 2008
commit  5fcb044cf5b4c43bde858e59d3acc6de87b8092d
tree    4b9cf3452303dc5988b1fafd30ce369ad7398068
parent  eda049dfd6c600c99c1de34ec0240139dde742b1 
kill faulty logdir() wrapping
numerodix (author)
Fri Jun 27 10:10:49 -0700 2008
commit  eda049dfd6c600c99c1de34ec0240139dde742b1
tree    19866618abc4957393c53f62007a417808e9f3cf
parent  49f20e35e9b289d39cb8ea934db6de11f371e95f 
fix bugs that caused fetching to halt after two redirects
numerodix (author)
Fri Jun 27 09:03:23 -0700 2008
commit  49f20e35e9b289d39cb8ea934db6de11f371e95f
tree    efb98581205c636f656039d9043099889ce66db0
parent  87ad618134a043cb6400bdddcdb381664df9ba32 
SOCKET_TIMETOUT is env var
numerodix (author)
Fri Jun 27 07:30:27 -0700 2008
commit  87ad618134a043cb6400bdddcdb381664df9ba32
tree    cabc5b3c2b82e74207535263cc7e11c994f0ac06
parent  d8ec49d54707aff5119dc62a30413b513118d34c 
fix io.py sanity check, apply logdir() only on paths == basename
numerodix (author)
Fri Jun 27 07:28:26 -0700 2008
commit  d8ec49d54707aff5119dc62a30413b513118d34c
tree    0b0f47d4f68ba9540f6c060d140dfe504096f317
parent  e44f12958c4c4e5b31b7c2a596767879e5f106ec 
fix for short filenames that are null
numerodix (author)
Fri Jun 27 06:41:40 -0700 2008
commit  e44f12958c4c4e5b31b7c2a596767879e5f106ec
tree    907aaea97ed263082182d27bec1dfd2cd46ed0a8
parent  e5068d31e916afb58b044644040f1983dc3bb076 
require $TERM set to output color
numerodix (author)
Fri Jun 27 06:38:23 -0700 2008
commit  e5068d31e916afb58b044644040f1983dc3bb076
tree    abafd7437d8dd3a32bab14d54aaad4fe54c2ed9e
parent  17c7c8b61c88d8502d7aebcee2af0c69912dff96 
disable gzip, it's slow :(
numerodix (author)
Fri Jun 27 05:36:33 -0700 2008
commit  17c7c8b61c88d8502d7aebcee2af0c69912dff96
tree    3067b7a29963fd417f84e52cce8c621c81f2fd79
parent  e9464d66550733b307c1c75a1c69264d1f45eef4 
small fixes to fetch.py
numerodix (author)
Fri Jun 27 05:22:38 -0700 2008
commit  e9464d66550733b307c1c75a1c69264d1f45eef4
tree    df016db6243610cc625c28e8e059575858fbd6d7
parent  3e9bb240e2bcbf12ad1798646387e10d2c9a7bce 
extract queue splitting logic to split_queue. better, but still not pretty
numerodix (author)
Fri Jun 27 05:07:05 -0700 2008
commit  3e9bb240e2bcbf12ad1798646387e10d2c9a7bce
tree    2891632aa45ccb88a5940f7b5572c2ffe21b7a06
parent  f6e0031945fd9882ac14daa0a483d70f0b2c1bcc 
don't use 'tuple' as identifier
numerodix (author)
Fri Jun 27 04:10:09 -0700 2008
commit  f6e0031945fd9882ac14daa0a483d70f0b2c1bcc
tree    0e8c72c9c3218031e0f7e95ca3548bb9c0931c18
parent  16044c5b6927730ea8bafabe6b5a5fcd73e033ca

2008-06-26

minor
numerodix (author)
Thu Jun 26 19:14:34 -0700 2008
commit  16044c5b6927730ea8bafabe6b5a5fcd73e033ca
tree    d69fed0431e4a538684e7b7c9165417d5ca74163
parent  17e168841cad2cd9015c06854ac6024c1ca86ad7 
minor
numerodix (author)
Thu Jun 26 19:11:10 -0700 2008
commit  17e168841cad2cd9015c06854ac6024c1ca86ad7
tree    bfc640c73ce840441154047b822cbd3e9a214d65
parent  f0b16f45aa11013fdc88d4be2136a5e99c4069f9 
modularize and cleanup
numerodix (author)
Thu Jun 26 18:52:24 -0700 2008
commit  f0b16f45aa11013fdc88d4be2136a5e99c4069f9
tree    f910bcbd324b1c4defbfd251009ae746d7ae5fe5
parent  63ebf396c0ec7006095d3c5cf554ed8beae83e21 
extract url classifying logic to function
numerodix (author)
Thu Jun 26 16:33:32 -0700 2008
commit  63ebf396c0ec7006095d3c5cf554ed8beae83e21
tree    d88a48e0dd2d0c58503f5e91cafe94b4541eabf3
parent  5c68bd93977ec11a38a06892ea398f373492c2fe 
fix 503 logic
numerodix (author)
Thu Jun 26 15:35:22 -0700 2008
commit  5c68bd93977ec11a38a06892ea398f373492c2fe
tree    b7816e061af4da5d25ae35e7e02603e850124de3
parent  a1b75eacf9703c6597ac69fe3143ea70c0a0801f 
add logic for retrying on 503
numerodix (author)
Thu Jun 26 14:44:10 -0700 2008
commit  a1b75eacf9703c6597ac69fe3143ea70c0a0801f
tree    69017b4bccc3e52471f0ea63204d1d96613e3edb
parent  60f97aaec9e811ae95433e603b622910282b9971 
rewrite urls on redirect also
numerodix (author)
Thu Jun 26 12:39:35 -0700 2008
commit  60f97aaec9e811ae95433e603b622910282b9971
tree    8894eec0252477ee01fb723421e8008895afaa81
parent  d867a541a4326113e6cecfeb2e75a46f986207ac 
docs
numerodix (author)
Thu Jun 26 10:32:05 -0700 2008
commit  d867a541a4326113e6cecfeb2e75a46f986207ac
tree    18440cacea44d9308d24b5821a396ec062dc18fd
parent  ed6cb7eeacd43b731cc2aeda1ceaf421d6f9960f 
minor
numerodix (author)
Thu Jun 26 10:01:16 -0700 2008
commit  ed6cb7eeacd43b731cc2aeda1ceaf421d6f9960f
tree    75d472c4e08f13ecea36b7532d687fa33384ad23
parent  08d09ae37448cf6e97a50ac2c4834ff4dec9e7f9 
chmod +x
numerodix (author)
Thu Jun 26 09:55:33 -0700 2008
commit  08d09ae37448cf6e97a50ac2c4834ff4dec9e7f9
tree    c1b6e777c4ce88d0eef70da886b2943f0c2418d4
parent  151ade13aba034901dd6202aa332b5ec4fa39d0a 
remove cruft
numerodix (author)
Thu Jun 26 09:50:20 -0700 2008
commit  151ade13aba034901dd6202aa332b5ec4fa39d0a
tree    756c52fbcada1fea03b2da07c7a8a16995831f63
parent  dee0d2edc41de144eb89622091445e11c1beca8f 
Merge spiderfetch2 into python-rewrite
numerodix (author)
Thu Jun 26 09:37:33 -0700 2008
commit  dee0d2edc41de144eb89622091445e11c1beca8f
tree    1f257f7e77e2615e3629241d0c9b245f61bc0f0f
parent  7e2d2c817c936c8df00ab69e0445525fde3bcaa3 parent  d7eecc2ded2105de8cf971abc5e53309eea23fbd 
change videostream recipe to find urls at any depth
numerodix (author)
Thu Jun 26 08:35:52 -0700 2008
commit  d7eecc2ded2105de8cf971abc5e53309eea23fbd
tree    756c52fbcada1fea03b2da07c7a8a16995831f63
parent  4e440d3b47797627b2c7268990a3367bec95c247