Bdb frontier access

[csrster]

First I should explain that this pull request builds on the existing pull request #281 .

The request adds four new methods to the Frontier - both interface and implementation.

The methods are all accessors of various kinds and none of them are called from within Heritrix. Their purpose is to expose Frontier internals to querying and manipulation by external 3rd Party tools via the scripting console - so there should be very little risk that the changes will in themselves cause any instability.

The only non-trivial method of the four is exportPendingUris(). The idea of this method is to write all pending uris to a file-cache where a suitable tool can browse and search in them to determine which, if any, one might want to delete manually (after pausing the job).

These methods are taken directly from the NetarchiveSuite fork of Heritrix which means that the code has been tested at scale (with je 5.0.104) at both Netarkivet in Denmark and BnF.

[csrster]

The reason why we need the bdb/je upgrade is the bug described under point 2 here: https://download.oracle.com/otndocs/products/berkeleydb/html/je/je-5.0.104_changelog.html

I believe that what happens in older versions of je is that the cursor used in writing the frontier data to disk is not transactionally isolated from changes in the frontier as a result of ongoing harvesting, and this causes errors errors in bdb that can't be handled by the Java code.

[anjackson]

This LGTM and these localised changes shouldn't impact elsewhere.