kkrugler (Ken Krugler)

Force DiskQueue elements to implement Serializable

Added new approach to grouping/robots.txt processing.

Comment out failing test

Add new parseMeta field when created ParsedDatum

Use Tika's MediaType.parse to extract mime-type and charset.

Minor improvement to reporting cause of abort when mime-type isn't

Put Tika parser metadata into resulting ParsedDatum

Add support for parsed (Tika) metadata

Fixed up contrib so it builds

Removed code that worked around various Tika bugs, for bugs

Added support for setting baseUrl in FetchedDatum.

Moved Hadoop-specific support to new bixo.hadoop package.

Get rid of some big test data files we don't need

Get rid of two annotations that Eclipse doesn't like.

Fixed comment error.

Cleaned up FetchPipe tests

Fixed tests now that FetchPipe/FetcherQueue use DiskQueue and assume

Re-enable test now that TIKA-339 is fixed.

Fixed bug w/size() not including cached element.

Document min response rate

Fix up "helpful" contrib for latest Cascading/Bixo

First cut at Cascading filter that acts like the Debug filter but is tied

Added first cut at DiskQueue, which uses temp file to spill

Added a safeClose for output streams.

Protect against Hadoop 0.18.3 returning null when we

Minor comment cleanup

Modified testing tool to also attempt to parse the fetched content,

First step in optimizing the fetch buffering process -

Make sure the job doesn't get killed just because Tika throws

Use new Cascading support to tell the flow planner that

Use new Cascading support to tell the flow planner that

Don't throw exceptions for unknown HTTP status codes, as servers

Fix for Cascading 1.1

Switch to Cascading 1.1-WIP-84, since it fixes some issues with

Changed how we decide whether to abort the connection, to try

Added comments about problems I see with current FetcherManager

Remove debug logging for typical situation of getting status

Fixed bug w/not handling multiple ROBOTS attributes (comma separated).

Set the "Acccept" header to be mime-types we typically want, so we don't

Finished up adding support for aborting fetch due to invalid mime-type.

Use tighter bounds on HTTP fetching when getting robots.txt file:

Decreased default max retries from 20 to 10.

Improved documentation.

Also specify the Accept-Charset default header in HTTP requests.

Minor formatting

Add support for specifying the Accept-Language request header via FetcherPolicy.

Add -logger option, to specify name of appender to use

Comment out code that uses hdfs.DistributedFileSystem class, as

Work around FileOutputFormat.getTaskOutputPath being protected

Comment out call to get Reporter.getCounter, as that doesn't exist

Don't use BytesWritable.getBytes(), since that doesn't exist

Add support for setting appender to use via system property

Switch to using our own custom 0.19.2 AMI for Bixo.

Fix job jar task to use lib/ as directory name for dependent

Work around a bug (TIKA-339) that causes the wrong language to get

Ignore target/ directory, which is created if you run the "mvn project-info-reports:dependencies"

BIXO-86: Protect against DBZ exception with crawl delay is 0.

Go for 128K of robots.txt data, since some of these are > 64K (default).

Do better job of detecting getting back random HTML from a robots.txt request.