Enhancement: Improve the checksum for the `Dataset` (or the equality test)

[cwognum]

When a Dataset contains pointer columns, we compute the checksum based on the file path instead of the file content (see here). Since we currently use the checksum to determine equality between Dataset objects (see here), this can lead to a weird situation like the following:

src_dataset.to_json("/home/cas/Desktop")
dst_dataset = Dataset.from_json("/home/cas/Desktop/dataset.yaml")
src_dataset == dst_dataset  # This is False

How could we improve this system? @hadim mentioned using the number of bytes in a file as a proxy. That would work a lot better, but I don't think that will be performant enough as datasets get large and are stored remote.

[cwognum]

One improvement could be to update the dataset itself upon saving. Right now, saving a dataset does not affect the underlying data-structure that is being saved. You only see the changed upon loading again. Perhaps it makes more sense to update the dataset upon any operation that changes the paths in a pointer column, i.e. cache() and to_yaml().

[hadim]

Indeed this is not ideal. For hub-based objects that will be easy to solve with the uuid (in short, uniqueness logic is delegated to the hub and not the lib).

Maybe we should checksum the dataset file itself and add boolean flag is_large_dataset that will disable that but then the hash logic must be optional everywhere.

The file size is an alternative approach that could work actually and likely scale very well (I think it might be possible to pull the size of an object without having to download it on remote buckets).

---

I don't think this is a blocker for us moving forward at the moment. But we can keep that ticket open and revisit.

[cwognum]

See #29 (comment) for a relevant discussion on MD5 vs SHA256.

[cwognum]

@jstlaurent This seems relevant again given our discussion this morning.

Some additional comments:

• We will require a checksum for the Zarr directory to ensure data integrity when uploading to the Hub.
• While it is good to keep in mind that we eventually want to support TB-sized datasets (e.g. to inform the choice of hash function), let's focus on GB-sized ones for now.
• It's worth spending some time on this now. Once we start officially supporting Zarr datasets, it will become increasingly more difficult to change this without having to maintain multiple checksum implementations.

Our use case here is to check whether the entire Zarr dataset has been successfully uploaded. As long as this is not the case, a dataset will not be visible on the Hub. It would be great to provide the user with more detailed information on what went wrong during the upload and to only retry the necessary parts, but this use case could be supported with zarr.convenience.copy_all.

It seems we could use a solution based on zarr_checksum. I commented on zarr-developers/zarr-python#392 to explore adding this to the official Zarr repository.

This means the hash would change if we change the chunks or if change the encoding, but I believe this is not an issue for us.

[jstlaurent]

It seems we could use a solution based on zarr_checksum. I commented on zarr-developers/zarr-python#392 to explore adding this to the official Zarr repository.

I think we ended on the same issue while researching this. I wasn't sure how feasible it would be to use that lib. If it looks good, then that's a great head start.

This means the hash would change if we change the chunks or if change the encoding, but I believe this is not an issue for us.

I think that's fair to expect and indeed, not a problem for us.