Upgrade BDB JE to version 7.5.11 - IMPORTANT CHANGE

[anjackson]

For #277 and other issues relating to using a very old version of BDB JE, this patch updates to the latest version. Apart from one test which relied on Thread.interrupt() (which makes BDB flake out), the upgrade appears to be straightforward.

However, marking this as a Work In Progress until I get a chance to run a large-scale test with it.

[anjackson]

Small scale testing works fine and indicates that the .jdb file re-use problem in #277 is resolved by this change.

[anjackson]

Having tested this in production, it seems to work fine. The only issue was that that on-disk format has changed since 4.1.6. See https://docs.oracle.com/cd/E17277_02/html/changelog.html

Specifically, 4.1.6 databases require at least one manual step (see DbPreUpgrade_4_1 part of the changelog. I found this difficult to execute in concert with Heritrix3's checkpoint management.

So, the question is, do we want to try to support upgrading existing crawler state databases, or just say this is a new release means you need to wipe your state?

[ato]

We discussed this on the OH-SOS call and nobody present had a need to preserve state between crawler upgrades. It sounds reasonable to me that an upgrade will invalidate checkpoints. I guess if someone really needs to resume a pre-upgrade crawl they can still use the recover log right?

[anjackson]

That seems reasonable to me.

Have we heard from anyone from IA?

[nlevitt]

Hey, sorry. Andy brought this to my attention again on slack. Archive-It is coming up with a plan. I will alert other IA folks using heritrix.

[anjackson]

Okay, so I think we're good to go ahead here.