HubDatasetModuleFactoryWithParquetExport

[js2552]

What is the purpose of this class in the loader? It looks like it is intended to check if the auto-created parquet files exist, and if they do, use them to load the dataset instead of however the raw data was uploaded. However as best as I can tell it is not called anywhere in the loading process and so those checks never run. Is this powering the dataset viewer behind the scenes?

[sueun-dev]

yeah, you're reading it right. on datasets 4.5.0 HubDatasetModuleFactoryWithParquetExport is defined (load.py:713) but instantiated nowhere, dataset_module_factory only builds the Local / Packaged / Hub / Cached factories. checked it two ways: an AST scan of load.py shows 0 instantiations of it, and instrumenting a real load_dataset() only fires HubDatasetModuleFactory, never the parquet-export one.

what it's for: its get_module() pulls the repo's auto-converted Parquet export via _dataset_viewer.get_exported_parquet_files, so it'd load straight from that parquet instead of the original files. it's just not on the load path right now.

and it doesn't power the viewer, that's a separate server-side service. this class would be a consumer of the same parquet export, not the thing producing it.