Redundant hash value in metadata

[thclark]

Bug report

What is the current behavior?

A hash_value key occurs in object metadata, but this is redundant with GCS' own hash value, which is exposed as md5_hash on the blob.

What is the expected behavior?

We shouldn't have to copy the hash from the blob to the custom metadata for the octue system to work. This is particularly frustrating if we're retroactively applying octue metadata to files already in a store, or we're uploading to the store using file-types that aren't necessary local files on disk.

Your environment

• Library Version: 0.1.15
• Platform n/a

[cortadocodes]

The hash type we're using on GCP is the CRC32C hash (available in parallel with the MD5 hash). Both of these are hashes of just the file contents and not the metadata, but the hashes we calculate for Datafiles are for the file contents and the metadata. Additionally, the crc32c attribute of the storage blob has to be the the hash of just the file contents, or the upload fails (in order to check for file upload corruption).

I think we have three options:

1. Make Datafile.hash_value only be the hash of the file contents
2. Have a different value for blob.crc32c to Datafile.hash_value (i.e. generate Datafile.hash_value by combining blob.crc32c with the hash of the Datafile's attributes
3. Keep the current implementation where blob.crc32c is the hash of the file contents, Datafile.hash_value is the hash of the file contents plus the metadata, and Datafile.hash_value is stored in the custom metadata of the blob as hash_value

[thclark]

The hash type we're using on GCP is the CRC32C hash (available in parallel with the MD5 hash). Both of these are hashes of just the file contents and not the metadata, but the hashes we calculate for Datafiles are for the file contents and the metadata. Additionally, the crc32c attribute of the storage blob has to be the the hash of just the file contents, or the upload fails (in order to check for file upload corruption).

I think we have three options:

1. Make Datafile.hash_value only be the hash of the file contents
2. Have a different value for blob.crc32c to Datafile.hash_value (i.e. generate Datafile.hash_value by combining blob.crc32c with the hash of the Datafile's attributes
3. Keep the current implementation where blob.crc32c is the hash of the file contents, Datafile.hash_value is the hash of the file contents plus the metadata, and Datafile.hash_value is stored in the custom metadata of the blob as hash_value

We could go down a massive rabbithole. Putting my business hat on...

• nobody's paying us to keep detailed QC on metadata consistency,
• having the metadata contents encapsulated in hash_value is clearly creating a huge hole in the head
  • especially for cases where metadata gets edited by services outside of octue's purview, which is quite plausible and would create a headache for developers to maintain a correct hash value
  • especially if we're shifting to a key-value approach to storing the filterable tag items. (and thus no longer have a finite set of fixed keys to compute from)

So my recommendation here (now that I've fiddled with this from the other side of the desk, tagging data in windquest for consumption by octue services) is to stop hashing the metadata and have Datafile.hash_value be just the hash of the file contents.

Pragmatism over perfection!

[thclark]

Also: storing the hash of the metadata in the metadata might be a little hard to explain to a seasoned sceptic systems engineer!!

[cortadocodes]

Don't worry, the hash of the metadata doesn't include the hash of the hash :P

Ok cool. I'll stop hashing the metadata for Datafile, but what about Dataset and Manifest?

• A Dataset's hash is made up of the hash its metadata and its datafiles
• A Manifest's hash is mad eup of the hash of its metadata and its datasets

If we stop hashing metadata everywhere, we have consistency but lose some of the point of having hashes for Manifests and Datasets (although we'd still get a unique value reflecting the set of files contained).

Conversely, if we keep hashing attributes on Manifests and Datasets, then we get hashes that behave differently depending on whether they're for a Datafile or not.