Integrate hf datasets as a data source

[ParagEkbote]

Is your feature request related to a problem? Please describe.

The Hugging Face Hub recently surpassed 1M+ datasets. Many users store and share hf datasets on the Hub and apply it as a standard interface for accessing them. However, great_expectations does not currently provide Hugging Face datasets as a native datasource.

Describe the solution you'd like
Add support for Hugging Face datasets as a datasource in great_expectations. This would allow users to load datasets directly from the Hugging Face Hub and apply expectations to their data.

Describe alternatives you've considered
N.A.

Additional context
I would be interesting in contributing this integration, having done a similar integration contribution to dagster.

[ParagEkbote]

What do you think about the proposed integration?

cc: @joshua-stauffer

[joshua-stauffer]

Hi @ParagEkbote — thanks for reaching out and for offering to contribute this. A native Hugging Face integration could definitely be valuable for teams working with training, evaluation, and other AI/ML datasets.

Adding a new data source is an ongoing maintenance commitment for GX, so before we take it on I'd like two things in place:

• Community interest: I'd like to hear from other GX users who would use this integration, including the datasets and validation workflows they have in mind.
• A concrete design proposal (RFC): We need to agree on the scope and architecture before implementation begins.

I'll convert this issue to a discussion so the conversation stays open and others can weigh in.

The questions I'd most want an RFC to answer:

• What does a native data source give users over the already-working path of load_dataset(...).to_pandas() into a plain GX Pandas Data Source?
• How would Hugging Face's repository → configuration/subset → split → shard structure (with revision as a versioning axis across the repo) map onto GX's Data Source → Data Asset → Batch Definition → Batch model?
• Which Execution Engine would process the data? HF is a dataset registry and loading layer, not a compute backend; GX computes Metrics on Pandas, Spark, or SQL engines, so Pandas is the natural first target.
• What dataset types would the first version support? GX support for non-scalar data types is limited.
• How would the integration handle configurations, splits, pinned revisions, authentication, caching, and large datasets?
• Would streaming datasets (IterableDataset) be in or out of scope for v1?

Another lighter-weight approach would be a HF dataset docs recipe, similar to our existing Validate unstructured data tutorial.

Happy to keep exploring this with you through the discussion and RFC.

[ParagEkbote]

Thanks for reaching out about the same, here are my thoughts about the queries presented:

1. What does a native data source give users over load_dataset(...).to_pandas()?

The main proposition is making HF datasets first-class, reproducible GX assets instead of anonymous DataFrames. Today, users could write custom loading code before every validation and none of the dataset identity (repo, config, split, revision) survives into the validation results. A native source keeps that metadata attached, so validation results are traceable back to an exact dataset version, and it lets users validate directly from a dataset identifier.

2. Mapping HF concepts onto GX concepts

Proposed mapping:

Hugging Face	Great Expectations
Repository	Data Source
Configuration/subset	Data Asset
Split	Batch Definition parameter
Revision	Batch identifier/version
Shard	Batch slice / Batch parameters

This fits GX's existing model reasonably cleanly since GX Batches already represent versions or slices of data, becoming another batch-identifying parameter rather than a new concept.

3. Which Execution Engine?

Pandas could be a suitable option for the integration. HF datasets already expose Dataset.to_pandas(), which fits GX's existing Pandas execution engine with no new compute layer required. Support for PyArrow can be explored as well since HF datasets are traditionally saved as pyarrow objects.

4. Dataset types

v1 would support scalar/tabular types: numeric, string, boolean, categorical, timestamp. Unsupported types like image, audio or video would raise an informative validation error rather than silently failing.

5. Configurations, splits, revisions, auth, caching, large datasets

• Config/split/revision become datasource/asset parameters, passed through to load_dataset() as-is.
• Revision support would include branch, tag,and commit hash, with repo/config/split/revision recorded in the GX validation metadata for reproducibility.
• Auth: Reuse the HF hub loading directly (token= or HF_TOKEN).
• Caching: Fully delegated to datasets (HF_HOME, cache_dir).
• Large datasets: rely on Arrow-backed lazy loading with keep_in_memory=False, materializing only at batch-conversion time.

6. Streaming datasets (IterableDataset)

I would like to support streaming datasets for processing large datasets, but GX metrics (row counts, uniqueness, quantiles, completeness, distributions) assume random access or multi-pass computation, which IterableDataset's single-pass iterator model doesn't support. So the streaming support would be out of scope for v1 unless explicitly requested.

---

I'll convert this into a complete RFC after agreeing on the scope and architecture before implementation begins and with community interest.

cc: @joshua-stauffer