ac778d0b0700bb5296ba1eeaa28ecbdce2a5bd0c xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/ade2440311d3b33a8b9bd00d38e778d9b65c75ae ade2440311d3b33a8b9bd00d38e778d9b65c75ae 2008-09-20T15:06:52-07:00 2008-09-20T15:06:52-07:00 Fixed the bug for too long of lists to canonicalize. Script to run the application. c87f15a20a0f02e4882183795d9908354bd21b30 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 8b7ec6981bd7b4285d34cfaa505f8207764f32b5 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/ac778d0b0700bb5296ba1eeaa28ecbdce2a5bd0c ac778d0b0700bb5296ba1eeaa28ecbdce2a5bd0c 2008-09-20T14:13:58-07:00 2008-09-20T14:13:58-07:00 Integrating googleurl into main code path. e82493c1955befddf83affac47a4aa2de4918fb1 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 022513941de4c3fb03384ede436ec12caa3361b3 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/8b7ec6981bd7b4285d34cfaa505f8207764f32b5 8b7ec6981bd7b4285d34cfaa505f8207764f32b5 2008-09-17T21:39:39-07:00 2008-09-17T21:39:39-07:00 Including patch for the configure file. There is a bug in autoconf 2.62-2.63 disabling erlang macros. Including declaration of the fixed function. ea57f19a3ae43df0397ffc5b9d2e9fb1e32feff7 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 42a28ec8358e1166ce67a08be1f14d6b2f29a18a xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/022513941de4c3fb03384ede436ec12caa3361b3 022513941de4c3fb03384ede436ec12caa3361b3 2008-09-14T22:27:11-07:00 2008-09-14T22:27:11-07:00 Removed one of the macros for erlang that wasn't working and was causing problems on older Mac OS X boxes AC_ERLANG_SUBST_LIB_DIR bb4b9335c5975eeb95e38583949fa27e6552173f xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 946ebe187503bd16e71ba8c5fdd0d667e589e830 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/42a28ec8358e1166ce67a08be1f14d6b2f29a18a 42a28ec8358e1166ce67a08be1f14d6b2f29a18a 2008-09-14T22:01:33-07:00 2008-09-14T22:01:33-07:00 Changed format of erl_interface check 69e5ca1660cb4b1b059eafc735b9eaf37e19f4ba xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 3a73950ffdd60fb1b4ebc92683579bcb68c5fc98 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/946ebe187503bd16e71ba8c5fdd0d667e589e830 946ebe187503bd16e71ba8c5fdd0d667e589e830 2008-09-14T21:02:09-07:00 2008-09-14T21:02:09-07:00 Added mkinstalldirs, since I think this is supposed to be here for autoconf. f2a5253a80352d8c1b339f3557e94fe501b9178a xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 8618016ac50ba05243759d82a8a9251befcbd9a7 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/3a73950ffdd60fb1b4ebc92683579bcb68c5fc98 3a73950ffdd60fb1b4ebc92683579bcb68c5fc98 2008-09-14T20:57:43-07:00 2008-09-14T20:57:43-07:00 Hopefully no longer need headers from ICU, since we should be detecting the location of them in our configure.ac. 78c1da463cfa22915b34ec12d584987ffe0eb390 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local a4facf87d825378342fdfcac082393f6e598daa6 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/8618016ac50ba05243759d82a8a9251befcbd9a7 8618016ac50ba05243759d82a8a9251befcbd9a7 2008-09-13T23:28:33-07:00 2008-09-13T23:28:33-07:00 adding autogen.sh file 10bc8a39501d080a7a8497f613ccee1983516b6d xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 8046724daa3b4144d47880a1de5088e05e8bd6db xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/a4facf87d825378342fdfcac082393f6e598daa6 a4facf87d825378342fdfcac082393f6e598daa6 2008-09-13T23:15:00-07:00 2008-09-13T23:15:00-07:00 Added a macro to check that ICU is installed and get it's version 5dee08886419876ecd5a350668480a9bc5db78a1 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local f8cf3c46d96215cfa7edd2e0efff1d4ab7f83126 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/8046724daa3b4144d47880a1de5088e05e8bd6db 8046724daa3b4144d47880a1de5088e05e8bd6db 2008-09-13T22:57:32-07:00 2008-09-13T22:57:32-07:00 Detecting the version of erl_interface installed 3dc2911639e84e78f17b1b91c194c86d7ec2f67a xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local baa241c65b0802a7eb86031660014c02554fbee3 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/f8cf3c46d96215cfa7edd2e0efff1d4ab7f83126 f8cf3c46d96215cfa7edd2e0efff1d4ab7f83126 2008-09-13T22:41:54-07:00 2008-09-13T22:41:54-07:00 Added bin directory 72f109958624563bc64c34879a6714c887cae966 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 65deb45d27b455971f5926000b5a154554d1f5b5 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/baa241c65b0802a7eb86031660014c02554fbee3 baa241c65b0802a7eb86031660014c02554fbee3 2008-09-13T22:35:10-07:00 2008-09-13T22:35:10-07:00 Creates a directory if one doesn't exist, ideally 83be3859c7acc52d82606b054168df0ae07a3214 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 8da371e88f74b699bdf13e2e5ef514e248c56cb3 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/65deb45d27b455971f5926000b5a154554d1f5b5 65deb45d27b455971f5926000b5a154554d1f5b5 2008-09-13T22:29:50-07:00 2008-09-13T22:29:50-07:00 Deleting Makefile that's no longer used 317f173035dec1fbc1979ce165a5945c8e364182 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local efc00f4afda57d4ccdfc9cbb7aff04678e1faed3 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/8da371e88f74b699bdf13e2e5ef514e248c56cb3 8da371e88f74b699bdf13e2e5ef514e248c56cb3 2008-09-13T22:26:03-07:00 2008-09-13T22:26:03-07:00 Building up a more standard build system 0a7edbfc673a58cde97d51b9bbee4c1f676fc62c xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 4f853f3bd5bebf63d852807c0ad98b2190c67223 6a7f4753c528558b4920eca56a0dbdf24045c2f3 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/efc00f4afda57d4ccdfc9cbb7aff04678e1faed3 efc00f4afda57d4ccdfc9cbb7aff04678e1faed3 2008-09-13T19:21:25-07:00 2008-09-13T19:21:25-07:00 Merge commit 'origin' 448c3b48b6d9a1ab637cb892c204a47d0d6309ca xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 0efca38de222c2f5c27a8b85a48eaf147878d9fb xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/4f853f3bd5bebf63d852807c0ad98b2190c67223 4f853f3bd5bebf63d852807c0ad98b2190c67223 2008-09-13T19:09:36-07:00 2008-09-13T19:09:36-07:00 cleaning up e39b97f547aca160c90131eb7f2e9fd84a1fb7f7 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local e08bb70a247a8e4e9f2e2fc4666e205b409e75b7 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/0efca38de222c2f5c27a8b85a48eaf147878d9fb 0efca38de222c2f5c27a8b85a48eaf147878d9fb 2008-09-13T19:07:47-07:00 2008-09-13T19:07:47-07:00 Back up and running with canonicalize_links. c28c14f56844979d0262e293a21fe9f7b11ab334 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 4a069cf7d50c780f09fad805dddeefaac4af04a4 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/e08bb70a247a8e4e9f2e2fc4666e205b409e75b7 e08bb70a247a8e4e9f2e2fc4666e205b409e75b7 2008-09-13T17:57:09-07:00 2008-09-13T17:57:09-07:00 adding googleurl.erl 4bc85584b53dfd0d1dd012f1c673077bfed76344 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 83c8512d2a8d8295401e1e218545479fff6b5c16 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local http://github.com/formido/spider/commit/4a069cf7d50c780f09fad805dddeefaac4af04a4 4a069cf7d50c780f09fad805dddeefaac4af04a4 2008-09-13T17:56:31-07:00 2008-09-13T17:56:31-07:00 Committing google url code. 46df7d829b032177fa67f30d64042fa99c9b9014 xanados bjterry@Benjamin-Terrys-MacBook-Pro-15.local 3b4ea08c5ef2dfe735c5db0b257eabbab69233ae Michael Terry formido@gmail.com http://github.com/formido/spider/commit/6a7f4753c528558b4920eca56a0dbdf24045c2f3 6a7f4753c528558b4920eca56a0dbdf24045c2f3 2008-09-11T00:22:39-07:00 2008-09-11T00:22:39-07:00 Requests and decodes zip, when available Also noticed that Mochiweb's html parser is disappointingly fragile, so I guess that won't work out long term. In the near term, we should probably move to a special purpose link extractor and start a module to reliably parse tag soup. 5a7617fe87322cf8fd383cbb02197cb8184ca389 Michael Terry formido@gmail.com b7a08398a89261d6570594500b13f7fee4d962d8 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/3b4ea08c5ef2dfe735c5db0b257eabbab69233ae 3b4ea08c5ef2dfe735c5db0b257eabbab69233ae 2008-09-09T23:23:22-07:00 2008-09-09T23:23:22-07:00 README update 0e5e56a7d514bc1773671442905b5bcb1a648204 Michael Terry formido@gmail.com 72d59fd6c875c54416e91cf24076a4f9a022fa8f Michael Terry formido@gmail.com http://github.com/formido/spider/commit/b7a08398a89261d6570594500b13f7fee4d962d8 b7a08398a89261d6570594500b13f7fee4d962d8 2008-09-09T23:19:17-07:00 2008-09-09T23:19:17-07:00 Destroy .DS_Store files fd630115ba64799cfbd1be59411c4fe47e9dddde Michael Terry formido@gmail.com c81bbd18d5c191701d915097ad04990c300dab4b Michael Terry formido@gmail.com http://github.com/formido/spider/commit/72d59fd6c875c54416e91cf24076a4f9a022fa8f 72d59fd6c875c54416e91cf24076a4f9a022fa8f 2008-09-09T23:16:49-07:00 2008-09-09T23:16:49-07:00 Do I need to actually git-rm it? 559c6c0ce4b4b7d1aae23fcbf4962e392dd0410a Michael Terry formido@gmail.com c5a72f0889d4f32f0d86615664511b2f38a8b7f1 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/c81bbd18d5c191701d915097ad04990c300dab4b c81bbd18d5c191701d915097ad04990c300dab4b 2008-09-09T23:15:34-07:00 2008-09-09T23:15:34-07:00 Update .gitignore to kill .DS_Store files 4a613ce327ca398d19ceff851afc3b606fa4bcfc Michael Terry formido@gmail.com 2d557c9333e2f68f807494ea4e831d9461391452 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/c5a72f0889d4f32f0d86615664511b2f38a8b7f1 c5a72f0889d4f32f0d86615664511b2f38a8b7f1 2008-09-09T23:07:33-07:00 2008-09-09T23:07:33-07:00 Update .gitignore to kill .DS_Store files ac822b62f06e92fac1e6b44e6b126f67400cd453 Michael Terry formido@gmail.com b70a5e79bdad3e435a11281c2418d3263ab89558 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/2d557c9333e2f68f807494ea4e831d9461391452 2d557c9333e2f68f807494ea4e831d9461391452 2008-09-09T23:06:29-07:00 2008-09-09T23:06:29-07:00 Update .gitignore to kill .DS_Store files dcd17bb93434cf21f270c133039afa9d2fbac0f6 Michael Terry formido@gmail.com e35ff5f42d8e56c0e8d98488ee1ae5fa31022ea6 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/b70a5e79bdad3e435a11281c2418d3263ab89558 b70a5e79bdad3e435a11281c2418d3263ab89558 2008-09-09T23:01:53-07:00 2008-09-09T23:01:53-07:00 Updated README formatting 1f3fa85326a263c4fd05b8a2967270169ee6d982 Michael Terry formido@gmail.com a01f6ccb6400d86851962b2f26fad184b83b9edd Michael Terry formido@gmail.com http://github.com/formido/spider/commit/e35ff5f42d8e56c0e8d98488ee1ae5fa31022ea6 e35ff5f42d8e56c0e8d98488ee1ae5fa31022ea6 2008-09-09T23:00:12-07:00 2008-09-09T23:00:12-07:00 Updated README formatting 82e85665b7dab96b0197891e4069a674739ae11c Michael Terry formido@gmail.com a22e4e622a748977119622692f2b552a097e96d5 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/a01f6ccb6400d86851962b2f26fad184b83b9edd a01f6ccb6400d86851962b2f26fad184b83b9edd 2008-09-09T22:52:26-07:00 2008-09-09T22:52:26-07:00 Added README 82849bf77d612301e6891e61e0c426136d4912f4 Michael Terry formido@gmail.com 027583b955e534bc680d558a53ebb6d65ff061c7 c00fc0043cd2f170415bde4985952b9417d9faa5 Michael Terry formido@gmail.com http://github.com/formido/spider/commit/a22e4e622a748977119622692f2b552a097e96d5 a22e4e622a748977119622692f2b552a097e96d5 2008-09-09T22:48:15-07:00 2008-09-09T22:48:15-07:00 Merge branch 'master' of git@github.com:formido/spider cc68724c9007a16f91fd9ef66e84614a267e2523 Michael Terry formido@gmail.com