(4.x.x) Deduplicating BLOB Store by adamretter · Pull Request #2314 · eXist-db/exist

adamretter · 2018-11-27T15:13:04Z

This PR provides a new Deduplicating BLOB Store to eXist-db, the design and new features are explained in my blog here: https://blog.adamretter.org.uk/blob-deduplication/

Binaries of eXist-db 4.5.0 patched with this new BLOB Store can be downloaded for testing from: http://static.adamretter.org.uk/blob-dedup/

NOTE: This PR increments the storage format versions of the collections.dbx and Journal files. So a full Backup and Restore is required from previous versions of eXist-db.

In addition this PR adds the following features:

Updates eXist-db to use the FasterXML Java UUID Generator.
For Binary Documents, the digest and file size are now available in the XML-RPC and XML:DB APIs.
For Binary Documents, the digest and file size now appear in the properties dialog of the Java Admin Client.
Adds Binary deduplication to database Backup and Restore, this is disabled by default but can be enabled via a flag or command line option. This option is also exposed in the Java Admin Client as a checkbox.
Adds the XQuery function util:binary-doc-content-digest which calculates a digest for a Binary Document. It is also optimised to just retrieve the digest if the digest type matches that used for Binary deduplication (e.g. BLAKE2B-256).

NOTE: This PR first requires:

adamretter · 2018-12-24T02:53:16Z

@dizzzz @wolfgangmm could you please review?

dizzzz · 2018-12-26T09:33:37Z

@adamretter yes I will do

dizzzz · 2018-12-26T11:42:12Z

src/org/exist/backup/CreateBackupDialog.java


    JComboBox collections;
    JTextField backupTarget;
+    JCheckBox deduplicateBlobs;


this probably needs some documentation

src/org/exist/backup/Main.java

src/org/exist/util/UUIDGenerator.java

dizzzz

the proof is in the pudding ; I am impressed by the work, there might be some documentation needed on the new backup parameter. I hope the performance is as good or better than it was. I shared ideas on (1) nr of files in a directory [performance, inodes] and (2) compression of the blobs and a concern regarding virus scanners on windows hosts (corporate policy)

dizzzz · 2018-12-26T11:58:30Z

one question: if one removes a blob file (e.g. virus scanner) is this detected? @adamretter

adamretter · 2018-12-26T13:46:43Z

@dizzzz If the file is not available an exception will be thrown and logged.

wolfgangmm · 2019-01-10T15:09:01Z

@adamretter could you rebase? Looks all good to me code wise, but would test some more.

dizzzz · 2019-01-10T19:44:38Z

The two pre-requisites are pulled in.....

adamretter · 2019-02-05T00:31:06Z

@wolfgangmm Okay it is now rebased as you requested.

joewiz · 2019-02-06T13:00:55Z

With approvals from Dannes and Wolfgang, we just need to address the failing appveyor tests.

dizzzz · 2019-02-06T19:09:49Z

in only one test scenario, an upload fails.... time out ; no existdb issue afai can see

joewiz · 2019-02-07T14:12:09Z

Do the appveyor failures in the other two ports also seem harmless? They were about unresolved dependencies on de.betterform#betterform-exist;5.1-SNAPSHOT-20160615. See #2460 (comment) and #2461 (comment).

joewiz · 2019-02-09T15:02:46Z

Looks like this needs to be rebased now that #2456 was merged first.

joewiz · 2019-02-09T21:23:46Z

I should note that the other ports (#2460 and #2461) have been merged (!), so at the moment our branches are not aligned. We should address this ASAP, especially before any releases.

dizzzz · 2019-02-09T21:38:34Z

I dont understand... all PRs should be in right now. Is this a GitHub issue?

joewiz · 2019-02-09T21:52:41Z

@dizzzz I am also unclear why this PR shows a conflict, while the already-merged associated ports (#2460 and #2461) do not. The conflict here, though, is certainly due to the switch in locations of the extensions in #2456 - but that also had ports (#2463 and #2464). Resolving the conflict should be a matter of moving them over to the location that #2456 put them in, I guess.

ljo · 2019-02-09T22:15:19Z

@joewiz yes, sorry, moving the files should do it. Regarding the appveyor failure it is not a code problem, but something in appveyor. We also had problems reported by travis which were not there on subsequent runs.

…lves the copy self-deadlock issue

…rmat as we now have the Blob Store instead of filesystem binary storage

…we now have the Blob Store instead of filesystem binary storage

…y default)

… erroring

…ocument

…-RPC and XML:DB APIs

…ment util:binary-doc-content-digest($binary-resource, $algorithm)

…he Java Admin Client

joewiz · 2019-02-12T03:36:36Z

@adamretter Will this help with the problem described in https://stackoverflow.com/q/54640371/659732? If so, we could chime in and say that the forthcoming release will include the feature...

adamretter · 2019-02-12T10:38:06Z

@joewiz Unrelated I am afraid.

joewiz · 2019-02-12T12:08:30Z

@adamretter Ok, thanks!

adamretter added enhancement new features, suggestions, etc. needs 5.x.x backport needs develop port labels Nov 27, 2018

adamretter requested review from dizzzz and wolfgangmm November 27, 2018 15:13

adamretter force-pushed the feature/binary-dedup-4.x.x branch from 0b7bb32 to 12b6754 Compare December 14, 2018 03:43

dizzzz reviewed Dec 26, 2018

View reviewed changes

src/org/exist/backup/Main.java Show resolved Hide resolved

dizzzz reviewed Dec 26, 2018

View reviewed changes

src/org/exist/util/UUIDGenerator.java Show resolved Hide resolved

dizzzz approved these changes Dec 26, 2018

View reviewed changes

duncdrum added the needs documentation Signals issues or PRs that will require an update to the documentation repo label Jan 5, 2019

adamretter force-pushed the feature/binary-dedup-4.x.x branch from 12b6754 to 15145e4 Compare February 5, 2019 00:30

This was referenced Feb 5, 2019

Deduplicating BLOB Store #2460

Merged

(5.0.0) Deduplicating BLOB Store #2461

Merged

adamretter removed needs 5.x.x backport needs develop port labels Feb 5, 2019

dizzzz approved these changes Feb 6, 2019

View reviewed changes

joewiz mentioned this pull request Feb 9, 2019

(4.x.x) Deduplicating BLOB Store #2466

Closed

adamretter added 20 commits February 10, 2019 13:01

[feature] First draft of a de-duplicating BLOB store

399bc10

[refactor] Second design iteration on a de-duplicating BLOB store, so…

f74f5d9

…lves the copy self-deadlock issue

[ignore] code cleanup

b16c575

[refactor] Switch over to the new BlobStore

8c669e9

[feature] Increment the version number of the collections.dbx file fo…

32f2c53

…rmat as we now have the Blob Store instead of filesystem binary storage

[refactor] Abstract the type for locating a document

b454c28

[ignore] Cleanup

c18828f

[feature] Add WAL to the BlobStore to make it crash safe

8e85011

[feature] Increment the version number of the Journal file format as …

f35f5f0

…we now have the Blob Store instead of filesystem binary storage

[feature] Tool for dumping a CSV of the Blob Store

783efd3

[feature] Update to FasterXML Java UUID Generator

0f69972

[feature] Add BLOB deduplication option to Backup/Restore (disabled b…

2d24430

…y default)

[ignore] code cleanup

dd258dc

[feature] Show raw binary as UTF-8 string in Admin Client rather than…

b5c5b19

… erroring

[feature] Facility for getting/calculating the checksum of a Binary D…

39606e0

…ocument

[feature] Expose binary document file size and digest through the XML…

a9b25d7

…-RPC and XML:DB APIs

[feature] Added XQuery function to retrieve a digest of a Binary Docu…

bfc7031

…ment util:binary-doc-content-digest($binary-resource, $algorithm)

[feature] Expose binary digest and size in the properties dialog of t…

98dcfee

…he Java Admin Client

[feature] Implement a zero I/O optimization for copying BLOBs

b002bd5

[feature] Optimise the copying of Binary documents to almost zero I/O

4ebaad2

adamretter force-pushed the feature/binary-dedup-4.x.x branch from 15145e4 to 4ebaad2 Compare February 10, 2019 12:15

duncdrum merged commit 98f4aa2 into eXist-db:develop-4.x.x Feb 10, 2019

adamretter deleted the feature/binary-dedup-4.x.x branch April 8, 2019 08:03

Uh oh!

Conversation

adamretter commented Nov 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamretter commented Dec 24, 2018

Uh oh!

dizzzz commented Dec 26, 2018

Uh oh!

dizzzz Dec 26, 2018

Choose a reason for hiding this comment

Uh oh!

adamretter Dec 26, 2018

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dizzzz left a comment • edited by adamretter Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dizzzz commented Dec 26, 2018

Uh oh!

adamretter commented Dec 26, 2018

Uh oh!

wolfgangmm commented Jan 10, 2019

Uh oh!

dizzzz commented Jan 10, 2019

Uh oh!

adamretter commented Feb 5, 2019

Uh oh!

joewiz commented Feb 6, 2019

Uh oh!

dizzzz commented Feb 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joewiz commented Feb 7, 2019

Uh oh!

joewiz commented Feb 9, 2019

Uh oh!

joewiz commented Feb 9, 2019

Uh oh!

dizzzz commented Feb 9, 2019

Uh oh!

joewiz commented Feb 9, 2019

Uh oh!

ljo commented Feb 9, 2019

Uh oh!

joewiz commented Feb 12, 2019

Uh oh!

adamretter commented Feb 12, 2019

Uh oh!

joewiz commented Feb 12, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

adamretter commented Nov 27, 2018 •

edited

Loading

dizzzz left a comment •

edited by adamretter

Loading

dizzzz commented Feb 6, 2019 •

edited

Loading