c4d1a34a2412b50f93404c24d1189d32e2a73108 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d34bb58f1531e0f93270bc9e74876cb30b1233c0 d34bb58f1531e0f93270bc9e74876cb30b1233c0 2008-07-20T15:44:03-07:00 2008-07-20T15:44:03-07:00 resume bug, insance variable queue not updated bbf9a83ee104afa6a44349638b540bf5d1b946db Martin Matusiak numerodix@gmail.com f7351b8cf64d054850fd470b6518a9754afd7e58 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c4d1a34a2412b50f93404c24d1189d32e2a73108 c4d1a34a2412b50f93404c24d1189d32e2a73108 2008-07-20T15:30:18-07:00 2008-07-20T15:30:18-07:00 session bugs cabebcc73b547b094ca4fd38420b70c77eb66b74 Martin Matusiak numerodix@gmail.com 99fd4dda0a7b288ee1804ec72f849e1f59639264 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/f7351b8cf64d054850fd470b6518a9754afd7e58 f7351b8cf64d054850fd470b6518a9754afd7e58 2008-07-20T15:07:03-07:00 2008-07-20T15:07:03-07:00 disable autocommit, completely kills performance 2ad299055df3f6ec40abc6fa68731f94493b8fe4 Martin Matusiak numerodix@gmail.com ab0b5cc58e54924f8af46286ddee20e23e069794 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/99fd4dda0a7b288ee1804ec72f849e1f59639264 99fd4dda0a7b288ee1804ec72f849e1f59639264 2008-07-20T14:47:31-07:00 2008-07-20T14:47:31-07:00 use fewer indexes 056573faa069d23c21870a51a6a82144e30a3834 Martin Matusiak numerodix@gmail.com c2ef9d9a0df0c2c3b18fc10e5036f25f25e3890c Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/ab0b5cc58e54924f8af46286ddee20e23e069794 ab0b5cc58e54924f8af46286ddee20e23e069794 2008-07-20T14:35:30-07:00 2008-07-20T14:35:30-07:00 kill temp commit 664aeaf4c0e8d2e1fba89075112c44cae6480485 Martin Matusiak numerodix@gmail.com b0118dd7b0ef354c893985ab82209041e71f695e Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/c2ef9d9a0df0c2c3b18fc10e5036f25f25e3890c c2ef9d9a0df0c2c3b18fc10e5036f25f25e3890c 2008-07-20T14:17:16-07:00 2008-07-20T14:17:16-07:00 comment e9b587fe95152cdc8e7235228a9d861e3680d6ae Martin Matusiak numerodix@gmail.com 1551682fb18fdc71009b5050b9caa339166a69e7 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b0118dd7b0ef354c893985ab82209041e71f695e b0118dd7b0ef354c893985ab82209041e71f695e 2008-07-20T14:15:22-07:00 2008-07-20T14:15:22-07:00 fix bad timing in SqliteWeb init ee63bd65f39804a73b63343c0abe7537322449b9 Martin Matusiak numerodix@gmail.com 097b4d47b06c0fc0fa50d280a758872fef65886d Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1551682fb18fdc71009b5050b9caa339166a69e7 1551682fb18fdc71009b5050b9caa339166a69e7 2008-07-20T14:09:41-07:00 2008-07-20T14:09:41-07:00 adapt spiderfetch to use SqliteWeb now 21074cb87f89082316e33e613f67cadf48152b7d Martin Matusiak numerodix@gmail.com b54c9190b14f549cea4c61e325195b62ad4ce0a6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/097b4d47b06c0fc0fa50d280a758872fef65886d 097b4d47b06c0fc0fa50d280a758872fef65886d 2008-07-20T13:47:41-07:00 2008-07-20T13:47:41-07:00 fix sql logic bug ed7ae0990172d7bca582521b0f0fa3ff758c40b1 Martin Matusiak numerodix@gmail.com 6091680a81271f91e1011d86ca3bb5de430f9716 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b54c9190b14f549cea4c61e325195b62ad4ce0a6 b54c9190b14f549cea4c61e325195b62ad4ce0a6 2008-07-20T13:23:48-07:00 2008-07-20T13:23:48-07:00 bad timing bug 7b0ec061c055337283d431a65825d3d76175ffd0 Martin Matusiak numerodix@gmail.com e19cf868d1463adadf6faa9b2fca6a571da7280f Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/6091680a81271f91e1011d86ca3bb5de430f9716 6091680a81271f91e1011d86ca3bb5de430f9716 2008-07-20T13:18:45-07:00 2008-07-20T13:18:45-07:00 check file exists now --test and --testmem b77d662ed98090e989c1833a2a513d5620f60585 Martin Matusiak numerodix@gmail.com 85eb6be0d276a58ce240051832864c4baa1c9a46 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e19cf868d1463adadf6faa9b2fca6a571da7280f e19cf868d1463adadf6faa9b2fca6a571da7280f 2008-07-20T12:56:45-07:00 2008-07-20T12:56:45-07:00 db logic almost done 4e8bd083b0dc264b56e6e8067bb62f64f4ef7d73 Martin Matusiak numerodix@gmail.com 2c2201f81e8136752e85b2095eb4133f60cdba70 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/85eb6be0d276a58ce240051832864c4baa1c9a46 85eb6be0d276a58ce240051832864c4baa1c9a46 2008-07-20T10:22:22-07:00 2008-07-20T10:22:22-07:00 perf 34409af33336b0e99fa478196b39af5e10b69718 Martin Matusiak numerodix@gmail.com e315f4ffb5afb3c8cfc9624464cf707ace991ed5 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2c2201f81e8136752e85b2095eb4133f60cdba70 2c2201f81e8136752e85b2095eb4133f60cdba70 2008-07-20T10:20:45-07:00 2008-07-20T10:20:45-07:00 finish web.root 07458fb34ed7703a5bab51b3bc4e769a3e788034 Martin Matusiak numerodix@gmail.com df367bb72183411733d9eff31e86463117c51a48 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/e315f4ffb5afb3c8cfc9624464cf707ace991ed5 e315f4ffb5afb3c8cfc9624464cf707ace991ed5 2008-07-20T09:21:01-07:00 2008-07-20T09:21:01-07:00 start sql 9553539ef123f25489b6cd355380fba54ac0a96d Martin Matusiak numerodix@gmail.com b5e4a4b530320579cbefa422456ad89876686020 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/df367bb72183411733d9eff31e86463117c51a48 df367bb72183411733d9eff31e86463117c51a48 2008-07-20T07:28:24-07:00 2008-07-20T07:28:24-07:00 init web, clean so far :) 48e34d1bff9c11d4a460a8a0d60b68f8e1b68c88 Martin Matusiak numerodix@gmail.com 2d4eb68132fc2310df83911695bc56f763d78f9a Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/b5e4a4b530320579cbefa422456ad89876686020 b5e4a4b530320579cbefa422456ad89876686020 2008-07-20T06:51:10-07:00 2008-07-20T06:51:10-07:00 comments 4fa44be7c944eeaf9d10a696294e36e24148192a Martin Matusiak numerodix@gmail.com 19f5f1bd571f96420f2c52b39735376dea673de4 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/2d4eb68132fc2310df83911695bc56f763d78f9a 2d4eb68132fc2310df83911695bc56f763d78f9a 2008-07-20T06:49:52-07:00 2008-07-20T06:49:52-07:00 move web save/restore to web module 6980540840ed62fef35da9e4a4e2e762a50ee244 Martin Matusiak numerodix@gmail.com 659ba2ca156192dc1a073619fd366ca8a69f53b9 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/19f5f1bd571f96420f2c52b39735376dea673de4 19f5f1bd571f96420f2c52b39735376dea673de4 2008-07-20T05:15:39-07:00 2008-07-20T05:15:39-07:00 organize 6d4f35a8054bd8b1bdc8de927ce00f263225d2db Martin Matusiak numerodix@gmail.com bb46140e3e587706df3251ea794628b331b4d590 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/659ba2ca156192dc1a073619fd366ca8a69f53b9 659ba2ca156192dc1a073619fd366ca8a69f53b9 2008-07-20T04:37:01-07:00 2008-07-20T04:37:01-07:00 revert att names to pre rename for compat ead0c7143f48ba106d384cc03a6b742e7e9ae3e1 Martin Matusiak numerodix@gmail.com 1151f0ab0b360b19b9140f4a9862d88850299ff8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/bb46140e3e587706df3251ea794628b331b4d590 bb46140e3e587706df3251ea794628b331b4d590 2008-07-20T04:32:20-07:00 2008-07-20T04:32:20-07:00 add nodeid att to Node in preparation for db subclassing a2ed131a2a80f06b9b48c60fabff9be1be2e3b88 Martin Matusiak numerodix@gmail.com 950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/1151f0ab0b360b19b9140f4a9862d88850299ff8 1151f0ab0b360b19b9140f4a9862d88850299ff8 2008-07-19T20:40:40-07:00 2008-07-19T20:40:40-07:00 api rewrite of web into small functional units for db subclassing 9da7c183ab11e2aecf85a2729640d3e9b948ee7b Martin Matusiak numerodix@gmail.com 62dd9e94779ca8fb259c69f29bdc737b156f3e16 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 950ce30d803a2fd3ff05f8e3c3bd8ee0631d1d34 2008-07-19T20:35:25-07:00 2008-07-19T20:35:08-07:00 prune unused gzip add header func 74c0fdec2fb0aeea2f377f8f19fad4aa4f404235 Martin Matusiak numerodix@gmail.com d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/62dd9e94779ca8fb259c69f29bdc737b156f3e16 62dd9e94779ca8fb259c69f29bdc737b156f3e16 2008-07-19T11:15:42-07:00 2008-07-19T11:15:42-07:00 make queue and wb instance variables 084e09bf07edf9f28098f2b2cb6761f7caf0d567 Martin Matusiak numerodix@gmail.com 554af5a465a020adf72903f094bd96987449fe10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 d9c40ef88dfd8d242dc54a4543e90e9bbf736dd8 2008-07-19T09:08:58-07:00 2008-07-19T09:08:58-07:00 wrap spiderfetch methods in class, manage state variables better bb3b948e4308b08593c22a951e896962188458c1 Martin Matusiak numerodix@gmail.com 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/554af5a465a020adf72903f094bd96987449fe10 554af5a465a020adf72903f094bd96987449fe10 2008-07-17T14:48:23-07:00 2008-07-17T14:48:23-07:00 spider.py also supports local files, include that in the help d164fbf3356e354a72da509b1714d55b36373dce Martin Matusiak numerodix@gmail.com 5d2f989aee8ea812b52c812aef5149f69c036a10 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 9f2c3df0c58f0d0dbd34cc408640a00666c41ec6 2008-07-17T11:37:59-07:00 2008-07-17T11:37:59-07:00 periodically save state 6f4782d744c4e5cc88be907d0857857a0e483923 Martin Matusiak numerodix@gmail.com 67359833ad7a2d8e2b71794902cc10ee77b43845 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/5d2f989aee8ea812b52c812aef5149f69c036a10 5d2f989aee8ea812b52c812aef5149f69c036a10 2008-07-17T10:56:58-07:00 2008-07-17T10:56:58-07:00 safer serialization by writing to .partial files 97f8d3e84d92035e6d8a1779fbf2ba4019aa4c0c Martin Matusiak numerodix@gmail.com 73432359c7508fb7b74ef08e16f0457ea81536d2 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/67359833ad7a2d8e2b71794902cc10ee77b43845 67359833ad7a2d8e2b71794902cc10ee77b43845 2008-07-09T14:57:56-07:00 2008-07-09T14:57:56-07:00 formatting 3181f7bf905381eeaffbd76a0878fc900344883a Martin Matusiak numerodix@gmail.com 1ff1a04b833f4e01fa530612c80ae73cde49d9a6 Martin Matusiak numerodix@gmail.com http://github.com/numerodix/spiderfetch/commit/73432359c7508fb7b74ef08e16f0457ea81536d2 73432359c7508fb7b74ef08e16f0457ea81536d2 2008-07-08T18:37:32-07:00 2008-07-08T18:37:32-07:00 avoid division by zero 1b35ae6c0328a873ad37656e587c930f537de4c0 Martin Matusiak numerodix@gmail.com