bundler/partitioner middleware

[jorgeorpinel]

See treeverse/dvc.org/issues/682 for context.

It seems like large data sets (in the TBs) tend to get bundled and/or partitioned in different ways and formats such as HDFS/HDF5/TFRecord files. This poses a challenge for DVC data versioning which calculates checksums at the file (or directory) level.

What would be the easiest way to extend DVC support for this kind of dataset storing practice? Perhaps a tool separate to DVC itself even, as some sort of middleware that enables transparency between the actual dataset, however it's organized into bundles and partitions, and DVC commands.

@jorgeorpinel , I guess one way would be to support chunking but not sure if HDFS has the API to interact with chunks instead of files

[jorgeorpinel]

It seems HDF5 supports chunking and there's a HDF5 Connector for Hadoop "to extract metadata and raw data from HDF5 and netCDF4 files on HDFS".

[efiop]

Closing as stale. HDFS supports such files, and provides built-in checksum, so dvc uses it with external outs/deps (though there are some limitations like checksum not being a hash).