kkrugler (Ken Krugler)

Improved error reporting while parsing robots.txt

Rolled in Vivek's mods to support language detection:

Moved ParseFunction in as a sub-class.

Clarified when skipped-by-scorer could be used

Make sure last status is set to current time versus 0.

Rename of pipe for clarity.

Fixed bug w/use of explicit map type for metadata in BaseDatum.

Added utility Splitter sub-assembly.

Support optional crawler version number.

Cleaned up constants

Added handy metadata constants.

Rolled in Vivek's diff that removes a bunch of warnings.

Added findAllSubdirs(), which can be used to build a combo set of pipes for processing

Try to re-add files with correct names

Arghhh - still trying to clean up rename mess

Move into main (got added to test by accident)

Change msg used during tagging of release.

Fixed some typos

Re-add files with the correct names

Try to clean up rename mess by getting rid of files with multiple cases.

Fixed build problem w/helpful project in contrib

Arghhh - still trying to clean up rename mess

Change msg used during tagging of release.

Fixed some typos

Try to clean up rename mess by getting rid of files with multiple cases.

Merge branch 'master' into release

Turned on case-sensitivity so that git "knows" about name changes

Added some documentation on the helpful contrib,

Added class for configuring Hadoop jobs (and some

Added Nutch's StringUtils, and pulled in our string

Log URL of item being parsed, at trace level.

When we get a fetch error, log at trace level.

Output extra info when a commonly failing integration test

Clean up name of FetchPipe content output tail pipe.

Bump log level to trace during testing

Update pom for helpful contrib to use 1.0-SNAPSHOT of bixo,

Added bixo.root.level property, so it's easier to override this

Output trace msg when generating a fetch list.

Output number of remaining URLs in status msg.

Trim the href URL we get back from the Tika parser, since it

Don't bother trying to resolve relative "about:" links.

Added test and fixed bug where domain name ended

Added serialization test.

Fixed bug where exception data (especially the msg) wasn't

Fixed bug that showed up when last header had key but no value

Update to release procedure.

More release procedure improvements.

Make names of util package classes consistent.

Minor tweak to order of steps for releasing.

First cut at command line tool that can be used to do

Use -crawldir vs. -outputdir to indicate directory that contains

Added UserAgent class, which is now used in place of full user-agent string.

Changed SiteCrawlerTest to be a long-running test, and improved

The HttpClient 3.1 code has to be there when running SimpleCrawlTool,

Include all dependent libraries when doing distribution build.

Get rid of unused test code.

Try to mask a bogus Hadoop map-reduce warning.

Fixed bug where check for JavaScript: and such was

Removed addition of "www." to raw (paid level) domains,

Reworked SimpleCrawlTool to be a site crawler that can

Minor naming cleanup

Add explicit check for sitemap: directive, which

Fixed a bug where we were setting the wrong UrlStatus when a

Add utility for injecting new meta-data value.

Added some documentation on the helpful contrib,

Added class for configuring Hadoop jobs (and some

Added Nutch's StringUtils, and pulled in our string

Log URL of item being parsed, at trace level.

When we get a fetch error, log at trace level.

Output extra info when a commonly failing integration test

Clean up name of FetchPipe content output tail pipe.

Bump log level to trace during testing

Update pom for helpful contrib to use 1.0-SNAPSHOT of bixo,

Added bixo.root.level property, so it's easier to override this

Output trace msg when generating a fetch list.

Output number of remaining URLs in status msg.

Trim the href URL we get back from the Tika parser, since it

Don't bother trying to resolve relative "about:" links.

Added test and fixed bug where domain name ended

Added serialization test.

Fixed bug where exception data (especially the msg) wasn't

Fixed bug that showed up when last header had key but no value

Update to release procedure.

Merge branch 'master' into release

Fixed bug where connection wasn't being aborted when we got back

If the score was set to the special "skip me" value, then

If we get an exception while loading robots.txt, don't kill the

Add support for getting the tail pipe name.

Added test for robots.txt redirecting to https, and getting an error fetching that.

Update status whenever number of URLs/domains being fetched changes.

Output HTTP status code & headers w/getMessage() call.

Added SKIPPED_FILTERED in preparation for better filtering support.

IScoreGenerator.generateScore no longer throws an IOException

Improve output when we get a fetch exception.

Add status time to datum.

Fixed Javadoc comment

Pulled out the RedirectResponseHandler and made it more general for use

Added explicit UNSET_CRAWL_DELAY to differentiate from the default crawl delay,

Added trust manager (from Nutch) needed for no-certificate SSL (https) connections.

Fixed up comments.

Minor update to release procedure

If the score was set to the special "skip me" value, then

If we get an exception while loading robots.txt, don't kill the

Add support for getting the tail pipe name.

Added test for robots.txt redirecting to https, and getting an error fetching that.

Update status whenever number of URLs/domains being fetched changes.

Output HTTP status code & headers w/getMessage() call.

Added SKIPPED_FILTERED in preparation for better filtering support.

IScoreGenerator.generateScore no longer throws an IOException

Improve output when we get a fetch exception.

Add status time to datum.

Fixed Javadoc comment

Pulled out the RedirectResponseHandler and made it more general for use

Added explicit UNSET_CRAWL_DELAY to differentiate from the default crawl delay,

Added trust manager (from Nutch) needed for no-certificate SSL (https) connections.

Fixed up comments.

Minor update to release procedure

Move tool out of test.

Support 1..n URLs in command line.

Fixed handler used for redirection test.

Fixed generation of redirected URL

Improve output of toString()

Get rid of release artifacts from master - these

Print extra info when we get a failure.

Cleaned up vestigial use of "testcase".

Comment out failing test for now.

Doesn't need the simulation web server.

Rename long-running tests to xxxLRTest, so we can key off that

Add step re getting rid of old distributions.

Clean up pom.xml

Update release info

We no longer depend on ICU4J, since we don't directly include the

More the "no domain" test to integration tests, since

Get rid of huge DMOZ file so our dist build is more reasonable.

Updated release procedure doc to match new reality.

Set version to 1.0-SNAPSHOT in master

Clean up code to use one collector for regular/error cases, since we don't

Bump up max redirects and retries.

Minor name cleanup

Support specifying the max retry count.

Add log4j.properties to src/main/resources so that it's possible to control

handle UrlStatus coming back (vs. raw string) from

Get rid of support for stuffing FetcherPolicy into properties - the

Add isAllowed(URL) method that never throws a checked exception.

Use UrlStatus.FETCHED vs. funky string for valid fetches.

No longer return boolean from offer, as it can't ever be used

The only current reason for an aborted fetch exception is

Added new SKIPPED_XXX statuses (stati?)

Added additional helper constructor.

Use extra funky values for NO_XXX constants, so that using 0

Added info about & license for DMOZ data.

Moved test into integration area, as it depends on external (DNS) resources.

Move tool out of test.

Fixed handler used for redirection test.

Fixed generation of redirected URL

Improve output of toString()

Get rid of release artifacts from master - these

Print extra info when we get a failure.

Doesn't need the simulation web server.

Rename long-running tests to xxxLRTest, so we can key off that

Add step re getting rid of old distributions.

Clean up pom.xml

Update release info

We no longer depend on ICU4J, since we don't directly include the

More the "no domain" test to integration tests, since

Get rid of huge DMOZ file so our dist build is more reasonable.

Updated release procedure doc to match new reality.

Set version to 1.0-SNAPSHOT in master

Rolled in HTML test.

Get rid of trailing space that was creating a test failure.

Switch to SimpleParser (wrapper for Tika) from the Nutch HTML parser.

Hide HttpClient 3.1 wire logging, which shows up when

Handle things like mailto: and javascript: in links, by

Get rid of now unused Nutch parsing code, since we're using Tika.

Fixed up datum creation

Fixed up datum creation

Added safety checks for creating datum with null values.

Rolled in HTML test.

Get rid of trailing space that was creating a test failure.

Switch to SimpleParser (wrapper for Tika) from the Nutch HTML parser.

Hide HttpClient 3.1 wire logging, which shows up when

Handle things like mailto: and javascript: in links, by

Get rid of now unused Nutch parsing code, since we're using Tika.

Fixed up datum creation

Fixed up datum creation

Added safety checks for creating datum with null values.

Rename HtmlParser to NutchHtmlParser.

Minor tweak to allow relative URLs in <base> tag, even if

Switch to using our patched version of Tika, which has been

Add support for "install" ant target that installs resulting jar