ARROW-8318: [C++][Dataset] Construct FileSystemDataset from fragments

[fsaintjacques]

• Simplified FileSystemDataset to hold a FragmentVector. Each Fragment must be a FileFragment and is checked at FileSystemDataset::Make. Fragments are not required to use the same backing filesystem nor the same format.

• Removed FileSystemDataset::format and FileSystemDataset::partitions.

• Since FileInfo is not required by neither FileSystemDataset and FileSystemDatasetFactory, it is possible to create a dataset without any IO involved.

• Re-introduced the natural behavior of creating FileFragment with their full partition expressions instead of removing the ancestors common partitions.

• Added Expression::IsSatisfiableWith method.

• Added missing compression cmake options to archery.

• Ensure FileSource holds a shared_ptr pointer. This is required to refactor FileSystemDataset to support Buffer FileSource and heterogeneous FileSystems.

• Rename type to id, following other classes.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-8318

[jorisvandenbossche]

Fragments are not required to use the same
backing filesystem nor the same format.

Shouldn't we require that? That seems the goal of UnionDataset to combine datasets with different formats

[fsaintjacques]

Shouldn't we require that? That seems the goal of UnionDataset to combine datasets with different formats

Maybe, this is still enforced if you use the Factory, but you can create a FileSystemDataset with fragments from different file format.

[bkietz]

This is looking good, thanks!

[bkietz]

Follow up for sub tree pruning https://issues.apache.org/jira/browse/ARROW-8658

[fsaintjacques]

Due to R failure (that I didn't catch because my installation was broken and using and old version of arrow), I'll revert the FileSystemDataset::format and make sure they're homogeneous.

[jorisvandenbossche]

From a user perspective, I find that also an added convenience. In Python, the FileSystemDataset.format attribute let you check the format of your dataset (instead of needing to check next(dataset.get_fragments()).format, which is not impossible of course, but I was using the format attribute in my dask branch).

[kszucs]

Fragments are not required to use the same backing filesystem nor the same format.

This makes me wonder, why do we need FileSystemDataset and/or UnionDataset at all? Because all of the filesystem and format specific logic is implemented by the discovery objects?

It implies that we could define dataset as a vector of arbitrary fragments.

BTW this was my conclusion after my python dataset prototype, but convinced myself that a generic dataset would prevent us implementing further optimizations.

[jorisvandenbossche]

Do we need FileSystemDataset, maybe not. Is it still useful, IMO yes.

As mentioned above, I personally find it convenient to know that my dataset has a single format / filesystem, and be able to easily check this format.
Now of course, it might be that this convenience is not worth it given the added complexity to the code. Or that such convenience could also be given by the wrapper languages (eg in Python we could still have a Dataset subclass for single formats)

[bkietz]

WRT convenient single format or single file system datasets, it would be straightforward (and possibly more useful) to provide accessories for subsets, Dataset.subsets(by_format=True): { format: subset ... } Or similar. IMHO the convenience added does not justify a whole class, at least in C++. As you noted even if we remove it in C++ it could still be a convenience subclass in Python

[fsaintjacques]

The ParquetFileSystemDataset will hold a parquet::Metadata for example.

[kszucs]

I'd like elaborate a bit more on the generic dataset class regardless what kind of wrappers do we provide.

• Do you plan to unify the filesystem classes into a single one which can contain any kind of fragments (including InMemoryFragment)?
• Will the UnionDataset still be required?