(4.x.x) Deduplicating BLOB Store

[adamretter]

This PR provides a new Deduplicating BLOB Store to eXist-db, the design and new features are explained in my blog here: https://blog.adamretter.org.uk/blob-deduplication/

Binaries of eXist-db 4.5.0 patched with this new BLOB Store can be downloaded for testing from: http://static.adamretter.org.uk/blob-dedup/

NOTE: This PR increments the storage format versions of the collections.dbx and Journal files. So a full Backup and Restore is required from previous versions of eXist-db.

In addition this PR adds the following features:

• Updates eXist-db to use the FasterXML Java UUID Generator.
• For Binary Documents, the digest and file size are now available in the XML-RPC and XML:DB APIs.
• For Binary Documents, the digest and file size now appear in the properties dialog of the Java Admin Client.
• Adds Binary deduplication to database Backup and Restore, this is disabled by default but can be enabled via a flag or command line option. This option is also exposed in the Java Admin Client as a checkbox.
• Adds the XQuery function util:binary-doc-content-digest which calculates a digest for a Binary Document. It is also optimised to just retrieve the digest if the digest type matches that used for Binary deduplication (e.g. BLAKE2B-256).

NOTE: This PR first requires:

1. #2305
2. #2310

[adamretter]

@dizzzz @wolfgangmm could you please review?

[dizzzz]

@adamretter yes I will do

[dizzzz]

this probably needs some documentation

[adamretter]

Sure.

[dizzzz]

the proof is in the pudding ; I am impressed by the work, there might be some documentation needed on the new backup parameter. I hope the performance is as good or better than it was. I shared ideas on (1) nr of files in a directory [performance, inodes] and (2) compression of the blobs and a concern regarding virus scanners on windows hosts (corporate policy)

[dizzzz]

one question: if one removes a blob file (e.g. virus scanner) is this detected? @adamretter

[adamretter]

@dizzzz If the file is not available an exception will be thrown and logged.

[wolfgangmm]

@adamretter could you rebase? Looks all good to me code wise, but would test some more.

[dizzzz]

The two pre-requisites are pulled in.....

[adamretter]

@wolfgangmm Okay it is now rebased as you requested.

[joewiz]

With approvals from Dannes and Wolfgang, we just need to address the failing appveyor tests.

[dizzzz]

in only one test scenario, an upload fails.... time out ; no existdb issue afai can see

[joewiz]

Do the appveyor failures in the other two ports also seem harmless? They were about unresolved dependencies on de.betterform#betterform-exist;5.1-SNAPSHOT-20160615. See #2460 (comment) and #2461 (comment).

[joewiz]

Looks like this needs to be rebased now that #2456 was merged first.

[joewiz]

I should note that the other ports (#2460 and #2461) have been merged (!), so at the moment our branches are not aligned. We should address this ASAP, especially before any releases.

[dizzzz]

I dont understand... all PRs should be in right now. Is this a GitHub issue?

[joewiz]

@dizzzz I am also unclear why this PR shows a conflict, while the already-merged associated ports (#2460 and #2461) do not. The conflict here, though, is certainly due to the switch in locations of the extensions in #2456 - but that also had ports (#2463 and #2464). Resolving the conflict should be a matter of moving them over to the location that #2456 put them in, I guess.

[ljo]

@joewiz yes, sorry, moving the files should do it. Regarding the appveyor failure it is not a code problem, but something in appveyor. We also had problems reported by travis which were not there on subsequent runs.

[joewiz]

@adamretter Will this help with the problem described in https://stackoverflow.com/q/54640371/659732? If so, we could chime in and say that the forthcoming release will include the feature...

[adamretter]

@joewiz Unrelated I am afraid.

[joewiz]

@adamretter Ok, thanks!