Skip to content

Conversation

@Wauplin
Copy link
Contributor

@Wauplin Wauplin commented Sep 14, 2023

This PR updates the "Data Persistence for Spaces" section (here and here) to showcase CommitScheduler instead of Repository with this example Space and this guide.

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Sep 14, 2023

The documentation is not available anymore as the PR was closed or merged.

Copy link
Contributor

@osanseviero osanseviero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure about this change. The schedule commit setup is nice but a bit complex to set up compared to just repository. Should we add a simple commit example to the guide?

@Wauplin
Copy link
Contributor Author

Wauplin commented Sep 14, 2023

The schedule commit setup is nice but a bit complex to set up compared to just repository.

Maybe it is the docs/Space that are not simple enough which makes it looks as a complicated setup. Because IMO it really isn't that complex to use CommitScheduler as long as you don't care about the data format (e.g. being compatible with dataset-preview). Complexity about the data format is the same with the scheduler or a git clone since it's a matter of "how to save the data".

Basically, the setup goes from

repo = Repository(
    local_dir="data/", clone_from=DATASET_REPO_URL, use_auth_token=HF_TOKEN
)
repo.push_to_hub()  # on every call

to

scheduler = CommitScheduler(
    repo_id=DATASET_ID, repo_type="dataset", folder_path="data/", token=HF_TOKEN
)

@osanseviero
Copy link
Contributor

Feel free to resolve the conflict and merge, thanks a lot!

@Wauplin
Copy link
Contributor Author

Wauplin commented Nov 16, 2023

Lol, thanks for reviving this PR @coyotte508!

@Wauplin Wauplin merged commit 641dea0 into main Nov 17, 2023
@Wauplin Wauplin deleted the update-dataset-storage-docs branch November 17, 2023 10:21
@Wauplin
Copy link
Contributor Author

Wauplin commented Nov 17, 2023

Conflict resolved, I'm merging. Thanks for the reminder!

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants