Orchestrating Hugging Face Datasets with Apache Airflow

[ParagEkbote]

I've worked on a contribution which integrates Hugging Face Datasets with Apache Airflow through a dedicated provider, bringing dataset operations into Airflow's orchestration model.

The provider ships with:

• Dataset operators: LoadDatasetOperator, MapDatasetOperator, StreamDatasetOperator, and PushToHubOperator.
• Hugging Face connection and hook: Centralized authentication and access to Hub APIs.
• Artifact management: Durable dataset artifacts with metadata manifests exchanged through XCom.
• Object storage integration: Support for local and remote storage backends through Airflow's ObjectStoragePath.
• OpenLineage integration: Dataset lineage and provenance tracking for Hugging Face repositories.

The integration is designed around familiar Airflow abstractions while retaining the underlying Hugging Face Datasets workflows.

A big thank you to Quentin Lhoest for his technical stewardship and guidance, Kaxil Naik for sponsoring and championing the integration and the wider Apache Airflow community for their support throughout the process.

Read the article: Load, Transform & Publish at Scale: Introducing the Hugging Face Provider for Apache Airflow