a3a12a0e9cb753917d92adad623d9524ac96a18a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d27633f9adc454954afec782743a0ccbe5714650 d27633f9adc454954afec782743a0ccbe5714650 2008-04-27T14:28:08-07:00 2008-04-27T14:28:08-07:00 add option to load local recipes f6d885679fb0066eae2c8825648c013a8ebfd70b Martin Matusiak numerodix@gmail.com 38fa3380171a0866cb2179a0da666d4af9f98c79 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/a3a12a0e9cb753917d92adad623d9524ac96a18a a3a12a0e9cb753917d92adad623d9524ac96a18a 2008-04-27T13:42:52-07:00 2008-04-27T13:42:52-07:00 prettify asx recipe 01d51442468883b719836c2f1444fe734d83d5be Martin Matusiak numerodix@gmail.com ebae3b2397b68ac726356ceb1b0e542f6abace51 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/38fa3380171a0866cb2179a0da666d4af9f98c79 38fa3380171a0866cb2179a0da666d4af9f98c79 2008-04-27T02:28:54-07:00 2008-04-27T02:28:54-07:00 don't decrement to infinity and beyond aecb51ca6d26ce5fe19c61fd9c297268b0681d62 Martin Matusiak numerodix@gmail.com 123dd44fe1cb5fcd3380a3975851a03ec64eff1f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/ebae3b2397b68ac726356ceb1b0e542f6abace51 ebae3b2397b68ac726356ceb1b0e542f6abace51 2008-04-26T12:11:24-07:00 2008-04-26T12:11:24-07:00 kill dead code c7e7b3f10605aeda959ad6522affdd12e16ff485 Martin Matusiak numerodix@gmail.com 47e50e139217c946f8fef978b8206e2f140c8570 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/123dd44fe1cb5fcd3380a3975851a03ec64eff1f 123dd44fe1cb5fcd3380a3975851a03ec64eff1f 2008-04-26T10:35:54-07:00 2008-04-26T10:35:54-07:00 allow regex objs in rules report empty url with use of --host 3e255a77c123ed6eeea141519d329f1cdc223860 Martin Matusiak numerodix@gmail.com 684b59175744fe814597eef4f92aaa391b0591b0 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/47e50e139217c946f8fef978b8206e2f140c8570 47e50e139217c946f8fef978b8206e2f140c8570 2008-04-26T09:54:11-07:00 2008-04-26T09:54:11-07:00 add host filter bd8df180ecb0ae5be84629216569d2bbb893702c Martin Matusiak numerodix@gmail.com b7b2d30fd909e3492b220b15344e57f407da7114 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/684b59175744fe814597eef4f92aaa391b0591b0 684b59175744fe814597eef4f92aaa391b0591b0 2008-04-26T09:27:57-07:00 2008-04-26T09:27:57-07:00 add --dump/--fetch override action in recipe bd52ba081dcb1a2dfa236c3b1683ab39d1876232 Martin Matusiak numerodix@gmail.com 7688d0b5a617fa831fc9948fda4a45f81f102959 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b7b2d30fd909e3492b220b15344e57f407da7114 b7b2d30fd909e3492b220b15344e57f407da7114 2008-04-26T09:13:28-07:00 2008-04-26T09:13:28-07:00 strict separation between status output (stderr) and product output (stdout) gracefully handle ctrl+c, fail without stopping on other download failures remove logfile, wget logging either to screen or none at all display spidering progress 4eefd032b068af1393ba1fe626039425ff628830 Martin Matusiak numerodix@gmail.com 2988faa7cd36e4c06543811ecd7aa7133cad98fb Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/7688d0b5a617fa831fc9948fda4a45f81f102959 7688d0b5a617fa831fc9948fda4a45f81f102959 2008-04-25T13:22:07-07:00 2008-04-25T13:22:07-07:00 comment big loop 4468ff3c20dbb5bdb7cb1ac3635e45e95790e47c Martin Matusiak numerodix@gmail.com 1b223686ef9f7d7de1236c77dd6f1deeddd6c336 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2988faa7cd36e4c06543811ecd7aa7133cad98fb 2988faa7cd36e4c06543811ecd7aa7133cad98fb 2008-04-25T12:44:47-07:00 2008-04-25T12:44:47-07:00 major rewrite to add recursive spidering 9f9659345ca5a478e875aa90455d83d75a7fe063 Martin Matusiak numerodix@gmail.com 4713a82e12bd480a31d515452552357fec952d37 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1b223686ef9f7d7de1236c77dd6f1deeddd6c336 1b223686ef9f7d7de1236c77dd6f1deeddd6c336 2008-04-25T08:42:12-07:00 2008-04-25T08:42:12-07:00 'enhance' user agent string a89089d3ce859c579d2e58fb58b40e249b1d56a5 Martin Matusiak numerodix@gmail.com 7833f95d594ad685c6b620a1040554630d8b264d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/4713a82e12bd480a31d515452552357fec952d37 4713a82e12bd480a31d515452552357fec952d37 2008-04-25T07:06:49-07:00 2008-04-25T07:06:49-07:00 modularize to prepare for recursive spidering e9297c190e7cd1e339d2d9d2b50b90914f4f3d51 Martin Matusiak numerodix@gmail.com 85082d3e4bdcef3dc2d69252d66567ca8aeede5c Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/7833f95d594ad685c6b620a1040554630d8b264d 7833f95d594ad685c6b620a1040554630d8b264d 2008-04-25T04:11:43-07:00 2008-04-25T04:11:43-07:00 multiline match for in_tag matching clarify meaning in format() extend testcase 7907d78f1457d7f5650defb379bdb53809f4ce84 Martin Matusiak numerodix@gmail.com f6af50ebbd0f61f925d010c434c0f98f08e0c382 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/85082d3e4bdcef3dc2d69252d66567ca8aeede5c 85082d3e4bdcef3dc2d69252d66567ca8aeede5c 2008-04-25T02:05:03-07:00 2008-04-25T02:05:03-07:00 remove unnecessary grouping f493ccbd6deb7bda585d367abffbec3af46b5606 Martin Matusiak numerodix@gmail.com bd5f4b48392a9413d5ee3de0391a0be88177e7db Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/f6af50ebbd0f61f925d010c434c0f98f08e0c382 f6af50ebbd0f61f925d010c434c0f98f08e0c382 2008-04-25T01:53:09-07:00 2008-04-25T01:53:09-07:00 fix nullpointer 22b0e22464d95afbd3dc5f99915ef9c1f1acb703 Martin Matusiak numerodix@gmail.com fd933b13faa2627e886d3aa386204d76c8c655ca Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/bd5f4b48392a9413d5ee3de0391a0be88177e7db bd5f4b48392a9413d5ee3de0391a0be88177e7db 2008-04-25T01:48:19-07:00 2008-04-25T01:48:19-07:00 hash based optional params 5112b49d62fb4a50b5aad67ee612267207bd6fc0 Martin Matusiak numerodix@gmail.com 6a076a9691661c07f5512c32ec34490984daa430 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/fd933b13faa2627e886d3aa386204d76c8c655ca fd933b13faa2627e886d3aa386204d76c8c655ca 2008-04-25T00:46:37-07:00 2008-04-25T00:46:37-07:00 limit list of regexs to 6 5ecdbfe099219e93a7999d1dd04bfda5f29e08cb Martin Matusiak numerodix@gmail.com c372bdd0300d7c27104cafa3fab8bfd2a8332ae8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/6a076a9691661c07f5512c32ec34490984daa430 6a076a9691661c07f5512c32ec34490984daa430 2008-04-25T00:32:22-07:00 2008-04-25T00:32:22-07:00 apply protocol filter on matches add regex from library 11645acbc784fdbd684e29abbbd423a22b93f5aa Martin Matusiak numerodix@gmail.com 614dad7d4dd19df8b1579b3147331cab5b6803f9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c372bdd0300d7c27104cafa3fab8bfd2a8332ae8 c372bdd0300d7c27104cafa3fab8bfd2a8332ae8 2008-04-24T16:39:58-07:00 2008-04-24T16:36:03-07:00 make sorting among equal matches deterministic, first regex to match gets inner wrap (and its color shown) 22d8f78380758ca1983a08b1cc798d34ebfcb845 Martin Matusiak numerodix@gmail.com 5bdff6fd1b4df8fd00e31ff54dd820f54768b8f7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/614dad7d4dd19df8b1579b3147331cab5b6803f9 614dad7d4dd19df8b1579b3147331cab5b6803f9 2008-04-24T16:10:22-07:00 2008-04-24T16:10:22-07:00 reinstate missing pattern check dbfe913342e1dceb90cb78b16eee3e9450985629 Martin Matusiak numerodix@gmail.com 0ae7b5c66d60b7d118a5059871a601a5179cadd4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5bdff6fd1b4df8fd00e31ff54dd820f54768b8f7 5bdff6fd1b4df8fd00e31ff54dd820f54768b8f7 2008-04-24T16:00:36-07:00 2008-04-24T16:00:36-07:00 kill dead code b246b7bbf20057a0c9d12aaa6caa309568dea74b Martin Matusiak numerodix@gmail.com d9e2ee10aa6a6c96570b43b3652cbacf3d735b69 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/0ae7b5c66d60b7d118a5059871a601a5179cadd4 0ae7b5c66d60b7d118a5059871a601a5179cadd4 2008-04-24T15:57:52-07:00 2008-04-24T15:57:52-07:00 fix highlight for overlapping match 9deb0ba6f5b578a8642c9f5008f4bb33e2abafcc Martin Matusiak numerodix@gmail.com 3dee1aabf6e8dd15628266291646e385d6c797e4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d9e2ee10aa6a6c96570b43b3652cbacf3d735b69 d9e2ee10aa6a6c96570b43b3652cbacf3d735b69 2008-04-24T15:41:51-07:00 2008-04-24T15:41:51-07:00 display matches in bold color when overlapping 0251f70cea499dcdfd8a83426a61ac7906d2a9d7 Martin Matusiak numerodix@gmail.com c79b4ea4ab4a5209b83fa5e1a72b888635b6e3fb Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/3dee1aabf6e8dd15628266291646e385d6c797e4 3dee1aabf6e8dd15628266291646e385d6c797e4 2008-04-24T15:16:41-07:00 2008-04-24T15:16:41-07:00 major rewrite to allow multiple regexs and match coloring d251c85b362b6a7a92fb41a91805883ec37eff19 Martin Matusiak numerodix@gmail.com 69598f64a5e560968d0629e767f0fe82a55ccbad Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c79b4ea4ab4a5209b83fa5e1a72b888635b6e3fb c79b4ea4ab4a5209b83fa5e1a72b888635b6e3fb 2008-04-24T10:38:42-07:00 2008-04-24T10:38:42-07:00 add option to match on "src" attributes in tags 69d03678687756105c1483f55e012489e3d4d47d Martin Matusiak numerodix@gmail.com 8a57b78f5987166a6c35b3277bbf09d2b26d288a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/69598f64a5e560968d0629e767f0fe82a55ccbad 69598f64a5e560968d0629e767f0fe82a55ccbad 2008-04-24T09:22:18-07:00 2008-04-24T09:22:18-07:00 add cert nocheck by default b3268fcb446475909f0e99f38b22a1c98748b27f Martin Matusiak numerodix@gmail.com 878aaf2aa23e98dfee61850337f1ca8b05f93102 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/8a57b78f5987166a6c35b3277bbf09d2b26d288a 8a57b78f5987166a6c35b3277bbf09d2b26d288a 2008-04-24T09:05:25-07:00 2008-04-24T09:05:25-07:00 unset user agent string for wget 10cf4ec5e2bf8d944047d1d08b63fd85790de908 Martin Matusiak numerodix@gmail.com d4f7f88c26d10236f5ecf879f0b11fd54854befc Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/878aaf2aa23e98dfee61850337f1ca8b05f93102 878aaf2aa23e98dfee61850337f1ca8b05f93102 2008-04-24T07:30:01-07:00 2008-04-24T07:30:01-07:00 add some test cases f71922b5ed7074b83505e7d4ece3532b656c4c0f Martin Matusiak numerodix@gmail.com d41c4843d57263e94b9b351e9669da1e0e583b37 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d4f7f88c26d10236f5ecf879f0b11fd54854befc d4f7f88c26d10236f5ecf879f0b11fd54854befc 2008-04-24T07:24:25-07:00 2008-04-24T07:24:25-07:00 quote vars in bash script 9ea467dcfd7db72641c86e484e384c91672bcc40 Martin Matusiak numerodix@gmail.com 748fd984289f8900791051a0b32ad76f724e8e5b Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d41c4843d57263e94b9b351e9669da1e0e583b37 d41c4843d57263e94b9b351e9669da1e0e583b37 2008-04-23T16:38:37-07:00 2008-04-23T16:38:37-07:00 ungeneralize exception catch 8ec334a6f99be6f2f68a03557560c3d4bee3b119 Martin Matusiak numerodix@gmail.com