f94101f635c5972275a2a0406e7288f85c50e2e6 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f6003043554ed41258037a8a8959161a704a4675 f6003043554ed41258037a8a8959161a704a4675 2009-12-14T14:59:04-08:00 2009-12-14T14:59:04-08:00 Use DiskQueue vs. List for ScoredUrlDatum objects. 8159f660f60cce679f11f014ea50ea26fc466252 Ken Krugler kkrugler@transpac.com 1985f776f8b23283908fc31ceacfbf0e9cacffc9 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f94101f635c5972275a2a0406e7288f85c50e2e6 f94101f635c5972275a2a0406e7288f85c50e2e6 2009-12-14T14:58:36-08:00 2009-12-14T14:58:36-08:00 Get rid of two annotations that Eclipse doesn't like. 202b7622ac56826e52673eedfadfbb191c958231 Ken Krugler kkrugler@transpac.com c7339210dd0e4ea49c9b7f2381e90d2c7a2541fc Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/1985f776f8b23283908fc31ceacfbf0e9cacffc9 1985f776f8b23283908fc31ceacfbf0e9cacffc9 2009-12-14T14:58:18-08:00 2009-12-14T14:58:18-08:00 Fixed comment error. 27d60c8b1fd1241c0c032cedfb43935238689d87 Ken Krugler kkrugler@transpac.com 920f23fbc1a9e065c1c1ed18ac3446330f734a56 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/c7339210dd0e4ea49c9b7f2381e90d2c7a2541fc c7339210dd0e4ea49c9b7f2381e90d2c7a2541fc 2009-12-14T14:58:07-08:00 2009-12-14T14:58:07-08:00 Cleaned up FetchPipe tests 06130ee0a1e07defd2e27c2f8da93d78703dda1e Ken Krugler kkrugler@transpac.com 1ce62281bb60d755e28a8151e382bde082cc3b4a Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/920f23fbc1a9e065c1c1ed18ac3446330f734a56 920f23fbc1a9e065c1c1ed18ac3446330f734a56 2009-12-14T14:57:38-08:00 2009-12-14T14:57:38-08:00 Fixed tests now that FetchPipe/FetcherQueue use DiskQueue and assume that the URLs are sorted by score. f62e3e4f16313a394241a317cf6e26a1d284a96b Ken Krugler kkrugler@transpac.com f7c6eccbfcd98c6f5a37f16ee8e4fd33142f4af6 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/1ce62281bb60d755e28a8151e382bde082cc3b4a 1ce62281bb60d755e28a8151e382bde082cc3b4a 2009-12-14T14:57:02-08:00 2009-12-14T14:57:02-08:00 Re-enable test now that TIKA-339 is fixed. f6a9f9866a3f2531f3153b2f2bd1c89b3b65c2b7 Ken Krugler kkrugler@transpac.com a19e5be543904c098681ca5b740a0f78babc216f Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f7c6eccbfcd98c6f5a37f16ee8e4fd33142f4af6 f7c6eccbfcd98c6f5a37f16ee8e4fd33142f4af6 2009-12-14T14:56:37-08:00 2009-12-14T14:56:37-08:00 Fixed bug w/size() not including cached element. Fixed bug w/not ensuring file is open before handling load request. Added buffering for object output stream, for efficiency during many writes. Added clear support, for efficiency when clearing out the queue. 652a4fb2ac371751bffa06c9213d97a4bd83684b Ken Krugler kkrugler@transpac.com 81a8173484aa1c08243d67aeea36d8a7418dbdd7 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/a19e5be543904c098681ca5b740a0f78babc216f a19e5be543904c098681ca5b740a0f78babc216f 2009-12-14T14:54:52-08:00 2009-12-14T14:54:52-08:00 Document min response rate ed12079b674749aa6fce9f270214b84075e806c2 Ken Krugler kkrugler@transpac.com a8cfc167e5bc6dce1c448974d18e5258ade7f132 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/81a8173484aa1c08243d67aeea36d8a7418dbdd7 81a8173484aa1c08243d67aeea36d8a7418dbdd7 2009-12-14T14:54:40-08:00 2009-12-14T14:54:40-08:00 Fix up "helpful" contrib for latest Cascading/Bixo b053361f513fae0c3464fb73fdc908a5d4731525 Ken Krugler kkrugler@transpac.com 3a8c919c8ae67d902f613eed24eb5dda296ad94e Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/a8cfc167e5bc6dce1c448974d18e5258ade7f132 a8cfc167e5bc6dce1c448974d18e5258ade7f132 2009-12-14T14:54:06-08:00 2009-12-14T14:54:06-08:00 First cut at Cascading filter that acts like the Debug filter but is tied into Log4J. 9b712713722cd75db169b1a23dff796986709e04 Ken Krugler kkrugler@transpac.com f9b42dfdae3859b1b7ddcde5dc3ec86583858da9 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/3a8c919c8ae67d902f613eed24eb5dda296ad94e 3a8c919c8ae67d902f613eed24eb5dda296ad94e 2009-12-11T17:22:40-08:00 2009-12-11T17:22:40-08:00 Comment out failing test w/Cascading 1.1 & Hadoop 0.18.3 d94a507d5e06476cd3f409dbc55b0c06c922d578 Ken Krugler kkrugler@transpac.com d8222ade5dbd31cd3e2bca6a693e412d59bd4206 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f9b42dfdae3859b1b7ddcde5dc3ec86583858da9 f9b42dfdae3859b1b7ddcde5dc3ec86583858da9 2009-12-11T17:22:08-08:00 2009-12-11T17:22:08-08:00 Added first cut at DiskQueue, which uses temp file to spill extra entries to disk. This will help us solve out of memory errors during queue generation for both robots.txt and FetcherBuffer processing. d16b6e4a13b6800f8df5fd0ba7cf44a4646238e9 Ken Krugler kkrugler@transpac.com 8aec1504729e194d6c1ed884191a759ba7bc9229 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/d8222ade5dbd31cd3e2bca6a693e412d59bd4206 d8222ade5dbd31cd3e2bca6a693e412d59bd4206 2009-12-11T17:21:09-08:00 2009-12-11T17:21:09-08:00 Added a safeClose for output streams. 19310e54778a981df7c6e67c9a30a28f3331d3e7 Ken Krugler kkrugler@transpac.com e19f2b508b3408a7671e5a0353c5f8a843cc709c Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/8aec1504729e194d6c1ed884191a759ba7bc9229 8aec1504729e194d6c1ed884191a759ba7bc9229 2009-12-11T17:20:57-08:00 2009-12-11T17:20:57-08:00 Protect against Hadoop 0.18.3 returning null when we ask for a list of sub-directories. In 0.19.x you get back an empty array of FileStatus objects. cd5b9815e0ed946468c6a0c8ea5de1e401aec844 Ken Krugler kkrugler@transpac.com 957495716bfeb7a870402872a8b0dd74ecdc7bd3 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/e19f2b508b3408a7671e5a0353c5f8a843cc709c e19f2b508b3408a7671e5a0353c5f8a843cc709c 2009-12-11T17:20:17-08:00 2009-12-11T17:20:17-08:00 Minor comment cleanup 70d75c00cde0eb6ebe97d9ee2788b42f7871f792 Ken Krugler kkrugler@transpac.com 386ccc208ac3bd79fd07a45210f97c21995c480c Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/957495716bfeb7a870402872a8b0dd74ecdc7bd3 957495716bfeb7a870402872a8b0dd74ecdc7bd3 2009-12-11T17:20:09-08:00 2009-12-11T17:20:09-08:00 Modified testing tool to also attempt to parse the fetched content, to debug what's happening with some funky documents we're getting back from Stanford.edu servers. 0d3f5068d036627a3b82a8b8b313837289005436 Ken Krugler kkrugler@transpac.com d579a0820fa201600759f784c14514b32c2f8037 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/386ccc208ac3bd79fd07a45210f97c21995c480c 386ccc208ac3bd79fd07a45210f97c21995c480c 2009-12-11T17:19:32-08:00 2009-12-11T17:19:32-08:00 First step in optimizing the fetch buffering process - sort by score (reverse order) before calling FetcherBuffer so that it can just start tossing URLs once it has enough. 3552287f2b17c480f3393c61f383972ffd08ad14 Ken Krugler kkrugler@transpac.com 7b37b8ef04ef7e0ac89b750230b3e8a71241cb7b Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/d579a0820fa201600759f784c14514b32c2f8037 d579a0820fa201600759f784c14514b32c2f8037 2009-12-11T17:18:53-08:00 2009-12-11T17:18:53-08:00 Make sure the job doesn't get killed just because Tika throws an Error (not an exception) during parsing. f8c4c66e210949a4e9ffc19a651d9459071d9da3 Ken Krugler kkrugler@transpac.com ed124899da44ca09e5edc95754f240eb6f9ed1bf Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/7b37b8ef04ef7e0ac89b750230b3e8a71241cb7b 7b37b8ef04ef7e0ac89b750230b3e8a71241cb7b 2009-12-11T17:18:28-08:00 2009-12-11T17:18:28-08:00 Use new Cascading support to tell the flow planner that we DO NOT want to be run twice, if a pipe split follows the parse sub-assembly. 56020165cd728a6fc2a38f4804c65f758f1c8afa Ken Krugler kkrugler@transpac.com 7477240da1784672b06263bf3ed2c40bd64e32cc Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/ed124899da44ca09e5edc95754f240eb6f9ed1bf ed124899da44ca09e5edc95754f240eb6f9ed1bf 2009-12-11T17:18:04-08:00 2009-12-11T17:18:04-08:00 Use new Cascading support to tell the flow planner that we DO NOT want to be run twice, if a pipe split follows the fetch sub-assembly. 7b933a0c979e6492fdc64bc556b817c3424e4caf Ken Krugler kkrugler@transpac.com 051b7910d3c77d03ed495eb1ece9c475fc4feb95 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/7477240da1784672b06263bf3ed2c40bd64e32cc 7477240da1784672b06263bf3ed2c40bd64e32cc 2009-12-11T17:17:28-08:00 2009-12-11T17:17:28-08:00 Don't throw exceptions for unknown HTTP status codes, as servers are free to return whatever they damn well please (like 999). dead3a6b185220cbbcad270559a1c9cbdeb616a7 Ken Krugler kkrugler@transpac.com 9bd3357ed79d0872e56dd7e6448bc88e4c625321 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/051b7910d3c77d03ed495eb1ece9c475fc4feb95 051b7910d3c77d03ed495eb1ece9c475fc4feb95 2009-12-11T17:17:00-08:00 2009-12-11T17:17:00-08:00 Fix for Cascading 1.1 b7ad8f68784fb92d73542d578593ebf1c211cedd Ken Krugler kkrugler@transpac.com ed03d02178f1244e4ded2fb7cb5024d5e276dad5 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/9bd3357ed79d0872e56dd7e6448bc88e4c625321 9bd3357ed79d0872e56dd7e6448bc88e4c625321 2009-12-11T17:16:33-08:00 2009-12-11T17:16:33-08:00 Switch to Cascading 1.1-WIP-84, since it fixes some issues with running in EMR (long file names), and has support for marking operations (like fetching & parsing) as "not safe", which means they won't be done twice due to a following split. 0574349593b0b3609c654c7da43a57d2f4f844ce Ken Krugler kkrugler@transpac.com ad89a30b669a57de51b3c959efd8b8c7ccda43b8 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/ed03d02178f1244e4ded2fb7cb5024d5e276dad5 ed03d02178f1244e4ded2fb7cb5024d5e276dad5 2009-12-07T00:09:28-08:00 2009-12-07T00:09:28-08:00 Remove noisy trace logging. Handle files that are UTF-8 and have a BOM. 960e9b0d7c0f948a28f2cb21338b8534e6196897 Ken Krugler kkrugler@transpac.com 9c6816e9e9dc0d73248aee65d03d1fd51b2ed7c1 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/ad89a30b669a57de51b3c959efd8b8c7ccda43b8 ad89a30b669a57de51b3c959efd8b8c7ccda43b8 2009-12-07T00:08:30-08:00 2009-12-07T00:08:30-08:00 Changed how we decide whether to abort the connection, to try to catch cases of not closing things properly. d51421a7f1890800158fbcf7d7690a1a58f7d4f7 Ken Krugler kkrugler@transpac.com f39ff6a57c47c71a45b82691e8b8da658fb11132 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/9c6816e9e9dc0d73248aee65d03d1fd51b2ed7c1 9c6816e9e9dc0d73248aee65d03d1fd51b2ed7c1 2009-12-07T00:07:52-08:00 2009-12-07T00:07:52-08:00 Added comments about problems I see with current FetcherManager architecture. 2412bb63cf2b2e83d040dfefa371b8e5e4b44108 Ken Krugler kkrugler@transpac.com a3264caef0791eff546b47ae164a23d4381f87f5 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f39ff6a57c47c71a45b82691e8b8da658fb11132 f39ff6a57c47c71a45b82691e8b8da658fb11132 2009-12-07T00:07:23-08:00 2009-12-07T00:07:23-08:00 Remove debug logging for typical situation of getting status codes in 4xx and 5xx range. 778661852cf1b5f1f83677079cdc508484c1c049 Ken Krugler kkrugler@transpac.com 8e102378bf8a70ed1b0bfca2751739e4740df4d5 Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/a3264caef0791eff546b47ae164a23d4381f87f5 a3264caef0791eff546b47ae164a23d4381f87f5 2009-12-06T19:03:38-08:00 2009-12-06T19:03:38-08:00 Fixed bug w/not handling multiple ROBOTS attributes (comma separated). For bullet-proofing, parse all attributes (break on comma). 14a52fd41b6d543638376223dad8f1c8f8b0a110 Ken Krugler kkrugler@transpac.com f2fadd1a91aea4f4a2531454a236a9ff61aa2ffd Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/8e102378bf8a70ed1b0bfca2751739e4740df4d5 8e102378bf8a70ed1b0bfca2751739e4740df4d5 2009-12-06T19:03:01-08:00 2009-12-06T19:03:01-08:00 Set the "Acccept" header to be mime-types we typically want, so we don't get a bunch that we can't handle. 330ece7b472b1da65d74f99f851632e7cede642a Ken Krugler kkrugler@transpac.com 0e4549507c6c55b7e1945c93713842bd00967b5f Ken Krugler kkrugler@transpac.com http://github.com/emi/bixo/commit/f2fadd1a91aea4f4a2531454a236a9ff61aa2ffd f2fadd1a91aea4f4a2531454a236a9ff61aa2ffd 2009-12-06T19:01:40-08:00 2009-12-06T19:01:40-08:00 Finished up adding support for aborting fetch due to invalid mime-type. 85cae52b6f625eb44a935001796cc92891f25c99 Ken Krugler kkrugler@transpac.com