Skip to content
Discussion options

You must be logged in to vote

Would any Hugging Face Datasets maintainers be willing to be technical stewards for this integration?

Happy to be a technical steward / co-maintainer :)

Which existing Datasets APIs should be prioritized, and are there compatibility considerations that should guide the design?

Maybe prioritize the high level load_dataset(), .map() and push_to_hub() APIs to start with and keep things simple ?
Then in terms of compatibility I'm not sure what are the main constains on the Airflow side, but since datasets relies on Arrow for storage and typing it's best if the implem is Arrow/parquet focused or compatible in terms of typing and data format

Actually I find your dagster integration really c…

Replies: 4 comments 9 replies

Comment options

You must be logged in to vote
0 replies
Answer selected by ParagEkbote
Comment options

You must be logged in to vote
2 replies
@ParagEkbote
Comment options

@vikramkoka
Comment options

Comment options

You must be logged in to vote
0 replies
Comment options

You must be logged in to vote
7 replies
@ParagEkbote
Comment options

@kaxil
Comment options

@ParagEkbote
Comment options

@lhoestq
Comment options

@ParagEkbote
Comment options

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Ideas
Labels
None yet
5 participants