RFC: Hugging Face Datasets Data Source for Great Expectations

[ParagEkbote]

Problem

Hugging Face Datasets provides a standardized loading interface for training, evaluation, and other machine-learning datasets. A dataset can be identified through a repository, configuration/subset, split, and revision, and can be loaded through the datasets library.

Today, a user can validate a Hugging Face dataset with GX by loading it manually and converting it to a Pandas DataFrame:

from datasets import load_dataset

dataset = load_dataset(
    "wikimedia/wikipedia",
    "20231101.en",
    split="train",
    revision="a1f724c09",
)

df = dataset.to_pandas()

and then providing that DataFrame to a GX Pandas Data Source.

This works, but the Hugging Face dataset identity and loading semantics remain outside the GX data-source model. The repository, configuration, split, and revision are handled by user code rather than being represented as part of the GX Data Source, Data Asset, Batch Definition, and Batch model.

As a result:

• dataset loading must be implemented separately from the GX validation configuration;
• dataset identity and revision are not naturally represented in GX validation metadata;
• reproducibility requires users to manually preserve the Hugging Face loading parameters;
• configuration and split selection are not represented as GX Batch Definitions;
• users cannot configure a Hugging Face dataset as a first-class GX datasource and reuse normal GX batching and validation workflows.

A native datasource would therefore provide value beyond the existing load_dataset(...).to_pandas() path by making Hugging Face datasets reproducible GX assets rather than anonymous DataFrames.

The proposed integration is specifically for the Hugging Face datasets library.

Proposed change

Add a Hugging Face Datasets Data Source backed initially by the existing PandasExecutionEngine.

The proposed GX mapping is:

Hugging Face concept	Great Expectations component
Repository + credentials	Data Source
Execution Engine parameters	Data Asset
Configuration / subset + split	Batch Definition
Revision	Batch Parameter

This follows the existing GX pattern where the Data Source owns the connection and credentials, the Data Asset contains execution-engine-specific loading parameters, and the Batch Definition identifies the logical slice of data that becomes a batch.

Impact on public API or architecture

The initial API would follow the pattern discussed in #11987:

Proposed Public API

context = gx.get_context()

hf_datasource = context.data_sources.add_pandas_hugging_face(
    name="hf_demo_data_source",
    token="${HF_TOKEN}",
    repository="wikimedia/wikipedia",
    hf_endpoint="https://huggingface.co",
)

hf_asset = hf_datasource.add_dataset_asset(
    name="hf_demo_asset",
    columns=["id", "text", "language"],
)

hf_split_bd = hf_asset.add_batch_definition_split(
    name="wikipedia_en_train",
    config="20231101.en",
    split="train",
)

batch = hf_split_bd.get_batch(
    batch_parameters={
        "revision": "a1f724c09",
    }
)

The same Data Asset could expose a Batch Definition representing an entire configuration:

hf_whole_bd = hf_asset.add_batch_definition_whole_config(
    name="wikipedia_en_all",
    config="20231101.en",
)

batch = hf_whole_bd.get_batch(
    batch_parameters={
        "revision": "a1f724c09",
    }
)

The exact method names are proposed APIs, and can be refined as needed.

The Hugging Face Data Source would own:

• repository identifier;
• authentication credentials;
• Hugging Face endpoint;
• the selected GX Execution Engine.

Data Source

The datasource would delegate dataset loading to the Hugging Face datasets library rather than implementing a separate loading or caching layer.

v1 would focus on tabular datasets containing scalar values supported by the existing Pandas/GX validation path, including:

• numeric values;
• strings;
• booleans;
• timestamps;
• categorical-compatible values.

Images, audio, video, tensors, and other complex/non-scalar values would not be a target of the initial implementation.

Unsupported types should result in a clear error.

Data Asset

The Data Asset would contain parameters required to materialize the dataset for the selected execution engine.

For the initial Pandas implementation, this could include options such as column selection:

columns=["id", "text"]

This keeps execution-engine-specific materialization parameters separate from the logical batch selection.

Batch Definition

The Batch Definition identifies which logical portion of the dataset is being validated.

For Hugging Face Datasets, configuration/subset and split belong here:

configuration = "20231101.en"
split         = "train"

This also allows the API to support Hugging Face split expressions where appropriate, such as:

train[:10%]
train+test

The initial implementation would not expose shards as a user-facing batching axis.

Execution Engine

The initial implementation would use the existing PandasExecutionEngine.

This avoids introducing a new compute backend and allows the datasource to reuse the existing GX metrics layer.

The datasource should nevertheless keep dataset acquisition sufficiently separate from execution that a future Arrow-based Hugging Face source can be introduced without redesigning the user-facing dataset model.

The Arrow/DuckDB architecture discussed in #12101 is therefore considered a future compatibility target, not a dependency of this proposal.

GX architecture

The integration adds a new datasource following the existing GX datasource model:

Hugging Face Data Source
│
├── Repository
├── Credentials
└── Pandas Execution Engine
        │
        ▼
    Data Asset
        │
        ├── Materialization parameters
        │
        ▼
    Batch Definition
        │
        ├── Configuration
        └── Split
                │
                ▼
          Batch Parameter
                │
                └── Revision

At runtime:

GX Batch
   │
   ▼
Hugging Face load_dataset(...)
   │
   ▼
HF Dataset
   │
   ▼
Pandas DataFrame
   │
   ▼
PandasExecutionEngine
   │
   ▼
Existing GX Metrics / Expectations

This means the initial implementation should not require changes to the metrics layer.

Metadata and reproducibility

The integration should preserve the dataset identity used to construct a batch:

repository
configuration
split
revision

This allows validation results to retain a direct relationship to the source dataset and its version.

This is a central reason for providing a native datasource rather than relying exclusively on load_dataset(...).to_pandas().

Dependencies

The Hugging Face datasets dependency should be introduced as an optional dependency for the integration.

Support for IterableDataset (Streaming Datasets) in v1

Streaming would be valuable for large datasets, but a Pandas-based execution path requires materialization, while many GX Expectations require operations that are not naturally single-pass. The initial implementation therefore targets map-style Dataset objects. Streaming remains an explicit future extension, potentially enabled by an Arrow execution engine.

WDYT?

cc: @joshua-stauffer

Alternatives considered

Alternatives considered

1. Continue using load_dataset(...).to_pandas()

This is the current working solution and has the lowest implementation cost.

However, it leaves Hugging Face dataset identity outside the GX datasource model. Repository, configuration, split, and revision remain user-managed loading parameters rather than GX-native batch metadata.

It also couples users to a manual conversion step before validation.

This proposal therefore does not attempt to replace the existing path; it provides a first-class abstraction for users who want Hugging Face datasets represented directly in GX.

2. Implement an Arrow/DuckDB datasource directly

An Arrow/DuckDB implementation could avoid Pandas materialization and provide a more efficient path for large datasets.

However, the corresponding Arrow/DuckDB execution architecture is not currently an established GX ExecutionEngine. RFC #12101 proposes such a columnar path but explicitly treats its components as separable and incremental.

Making that architecture a prerequisite for Hugging Face support would therefore couple two independent changes and substantially increase the scope of the integration.

The proposed RFC instead starts with the existing PandasExecutionEngine and leaves a clean extension point for a future implementation.

[ParagEkbote]

Hello,

Just following up on this RFC to see if there are any specific concerns or open questions that are currently preventing it from moving forward.

Happy to clarify on the proposal as well :)

cc: @joshua-stauffer

[joshua-stauffer]

hey @ParagEkbote, thanks for your patience here! My main concern is that I haven't seen any signal from the community that there is a broader need for this feature, per my comment here. Since maintainer bandwidth is inherently limited, community signal is a major deciding factor for prioritizing work. I'm moving this forward to final comment, and will accept or reject it in the next two weeks.

In the meantime, I'm still interested in hearing from the community. If you are a GX user and are interested in using this proposed data source, please comment here or upvote the RFC proposal.

[joshua-stauffer]

Hi @ParagEkbote, thanks again for the proposal and your patience. I've collected enough community feedback that I'm willing to move forward with this, so I'm accepting the RFC.

Next step would be to open an issue with the overall design, and sub issues for each PR you plan to open. Each subissue should have their own requirements/AC, and in scope/ out of scope, with links to sibling issues where possible. To avoid this getting stuck in review, let's aim to keep PRs small and focused. Functional changes and their tests should always land in the same PR.

Thanks! looking forward to shipping this feature 🚀