Repository navigation
Hugging Face Datasets Provider for Apache Airflow #8882
MotivationI would like to explore developing an Apache Airflow provider for Hugging Face Datasets, enabling users to integrate dataset loading, processing, and publishing workflows into their Airflow DAGs. Hugging Face Datasets has provided a unified interface for loading datasets from the Hub, processing them efficiently, and working with Arrow-backed data. Apache Airflow provides orchestration for scheduling and managing data pipelines. A dedicated provider could make it easier for users to orchestrate HF-centric workflows without having to maintain custom Airflow operators for common operations. Proposed scopeThe initial scope could include operators, hooks and other integration components for:
Maintenance and ownershipI would particularly appreciate input from the HF Dataset Maintainers on whether this integration would be useful to the ecosystem and whether members of the team would be interested in co-maintaining it. Specifically:
Proposed governance and next stepsApache Airflow documents its process for accepting new community providers in ACCEPTING_PROVIDERS.rst. That process includes identifying stewards, obtaining sponsorship from an Airflow committer, providing a working implementation and meeting testing/ documentation requirements. One of the main points to consider would be to propose the provider for inclusion in the Apache Airflow project similar to the dagster integration or maintain it as a third-party provider like the pyspark source. I would welcome feedback and design suggestions before committing to a particular implementation. cc: @lhoestq |
Replies: 4 comments 9 replies
Happy to be a technical steward / co-maintainer :)
Maybe prioritize the high level Actually I find your dagster integration really cool so we could try to match it too
I would let someone from Airflow let us know what's the best path. Though if we want to have something already maybe let's start with a third party demo so we can iterate and the demo can serve for the inclusion in Airflow |
|
+1 from the Airflow practitioner side — a few things that would make or break this in production pipelines: 1. Cache control on 2. Map 3. Execution isolation + deferrable pattern. 4. Record provenance in XCom. Resolved dataset revision/sha on load, commit sha on push — cheap to add, and it's what makes lineage and reproducing a run possible. Re: scope, I'd second starting with |
|
Hello, I have created a mvp repo for the provider based on the conversation held here: https://github.com/ParagEkbote/hf-datasets-airflow-provider/blob/main/README.md Similarly, I have managed to transform the glue dataset with some normalization with Airflow and the result can be pushed to hf hub within the DAG as well. Feel free to try out the examples as well and if any features/suggestions could be added, be sure to let me know :) cc: @lhoestq |
|
Hi @ParagEkbote, thanks for kicking this off, and @lhoestq for offering to steward it. Happy to sponsor this as an Airflow committer & PMC Member. On the path, I'd go for a community provider in the Apache Airflow repo rather than a third-party package. The process is in ACCEPTING_PROVIDERS.rst: two stewards (the two of you cover that), a sponsoring committer, a working implementation (your MVP repo counts), and a [DISCUSS] thread on the dev list. If there are no objections within 7 days, you open the PR and the provider starts in incubation. I am happy to work with you on both the PR and email to dev list too. Two suggestions before the DISCUSS thread:
|
Happy to be a technical steward / co-maintainer :)
Maybe prioritize the high level
load_dataset(),.map()andpush_to_hub()APIs to start with and keep things simple ?Then in terms of compatibility I'm not sure what are the main constains on the Airflow side, but since
datasetsrelies on Arrow for storage and typing it's best if the implem is Arrow/parquet focused or compatible in terms of typing and data formatActually I find your dagster integration really c…