Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1162 Reorder operations correctly in commit endpoint #1175

Merged
merged 10 commits into from
Nov 10, 2022

Conversation

Wauplin
Copy link
Contributor

@Wauplin Wauplin commented Nov 9, 2022

Fix #1162.

With this PR:

  • Order of the operations in the commit payload is the same as the input of create_commit(...). It was not the case before which causes trouble if someone wants to delete then create a file.
  • If 2 operations are modifying the same file, a warning is issued. Only case without warning is "delete a file then add this file" or "delete folder then add file that would have been in this folder".
  • CommitOperationAdd is validated at initialization (no need for .validate() or ._upload_info() calls anymore)

Warning: I made some breaking changes in validate_preupload_info, fetch_upload_modes and prepare_commit_payload. I did not care too much as those 3 methods are defined in a private module and used exclusively for the in create_commit . I checked for safety and it's not used anywhere in diffusers, transformers, skops and datasets.

Potential issue: if someone adds twice the same file, once a LFS content, once a regular content then the final upload mode will be the one from the second operation. Might cause issue with request payload or gitaly. Anyway, this is a weird case so let's not care of the user has an error. A warning message is triggered before upload so the user should notice the problem. (note: this issue was already existing before this PR)

(cc @severo who raised the issue)

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Copy link
Member

@coyotte508 coyotte508 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conceptual ACK

src/huggingface_hub/hf_api.py Show resolved Hide resolved
@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice, thanks!

@@ -46,3 +50,49 @@ def test_explicit_folder(self):
def test_is_folder_wrong_value(self):
with self.assertRaises(ValueError):
CommitOperationDelete(path_in_repo="path/to/folder", is_folder="any value")


class TestWarnOnOverwritingOperations(unittest.TestCase):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean class!

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@HuggingFaceDocBuilder
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@Wauplin
Copy link
Contributor Author

Wauplin commented Nov 10, 2022

Thanks for the reviews @coyotte508 @LysandreJik ! I'm merging :)

@Wauplin Wauplin merged commit 6cc3641 into main Nov 10, 2022
@Wauplin Wauplin deleted the 1162-reorder-operations-in-commit-endpoint branch November 10, 2022 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Send delete operations before add operations in create_commit
4 participants