transformers-cli: LFS multipart uploads (> 5GB) #8663

julien-c · 2020-11-19T16:14:30Z

Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs. This lets users upload large files >5GB 🔥.

Spec for LFS custom transfer agent is: https://github.com/git-lfs/git-lfs/blob/master/docs/custom-transfers.md

The PR introduces two commands to the CLI:

transformers-cli lfs-enable-largefiles ./path/to/repo

^ Do this once per model repo where you want to push >5GB files. It's documented in the error message you get if you just try to git push a 5GB file without having enabled it before.

transformers-cli lfs-multipart-upload

^ is the custom transfer agent itself. This is not meant to be called by the user, but by lfs directly.

Things to experiment with:

upload speed. Is it sufficient? Please comment with your upload speeds e.g. for https://huggingface.co/t5-3b

Experiment:

time git clone https://huggingface.co/t5-3b
cd t5-3b
git remote set-url origin https://huggingface.co/$USER/t5-3b-clone
# ^ After having created this model repo in your account
transformers-cli lfs-enable-largefiles .

git reset 5e0a32db352b33091ea9fb2f8d8782d47a505986 # go back to initial commit for lfs to reupload files
git add pytorch_model.bin
git commit -m "ok"
time git push

src/transformers/commands/lfs.py

@lhoestq

cc @lhoestq

This reverts commit d7e32c4.

mymusise · 2020-11-20T15:15:08Z

👍 nice job!

src/transformers/commands/lfs.py

patrickvonplaten · 2020-11-20T15:31:56Z

src/transformers/commands/lfs.py

+    def run(self):
+        local_path = os.path.abspath(self.args.path)
+        if not os.path.isdir(local_path):
+            print("This does not look like a valid git repo")


maybe better to raise an error?

patrickvonplaten · 2020-11-20T15:33:41Z

src/transformers/commands/lfs.py

+        return data
+
+    def __iter__(self):
+        yield self.read(n=4 * 1024 * 1024)


why 4 * 1024 * 1024?

magic value from @madlag

sgugger

Looking good!
On the nit side, since I'm a very annoying person, I'd very much like if we could consistently use f-strings which are prettier and faster.

sgugger · 2020-11-20T15:27:27Z

src/transformers/commands/lfs.py

@@ -0,0 +1,219 @@
+"""


Should we add a copyright at the top?

sgugger · 2020-11-20T15:28:16Z

src/transformers/commands/lfs.py

+``` [lfs "customtransfer.multipart"]
+
+path = /path/to/transformers/.env/bin/python
+
+args = -m debugpy --listen 5678 --wait-for-client /path/to/transformers/src/transformers/commands/transformers_cli.py
+lfs-multipart-upload ```


This part is weirdly formatted.

Yep – is there any way to tell our formatting tooling to not change lines which represent code or structured stuff?

sgugger · 2020-11-20T15:28:29Z

src/transformers/commands/lfs.py

+
+
+class LfsCommands(BaseTransformersCLICommand):
+    # Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs.


src/transformers/commands/lfs.py

src/transformers/hf_api.py

sgugger · 2020-11-20T15:32:06Z

src/transformers/hf_api.py

+        assert filetype in self.ALLOWED_S3_FILE_TYPES, "Please specify filetype from {}".format(
+            self.ALLOWED_S3_FILE_TYPES
+        )


Same as above.

sgugger · 2020-11-20T15:32:20Z

src/transformers/hf_api.py

+        assert filetype in self.ALLOWED_S3_FILE_TYPES, "Please specify filetype from {}".format(
+            self.ALLOWED_S3_FILE_TYPES
+        )
+        path = "{}/api/{}/listObjs".format(self.endpoint, filetype)


Same as above.

sgugger · 2020-11-20T15:32:30Z

src/transformers/hf_api.py

+        assert filetype in self.ALLOWED_S3_FILE_TYPES, "Please specify filetype from {}".format(
+            self.ALLOWED_S3_FILE_TYPES
+        )
+        path = "{}/api/{}/deleteObj".format(self.endpoint, filetype)


Same as above.

patrickvonplaten · 2020-11-20T15:36:07Z

src/transformers/hf_api.py

        """
        HuggingFace S3-based system, used for datasets and metrics.

        Get a presigned url, then upload file to S3.

        Outputs: url: Read-only url for the stored file on S3.
        """
-        urls = self.presign(token, filename=filename, organization=organization)
+        assert filetype in self.ALLOWED_S3_FILE_TYPES, "Please specify filetype from {}".format(


(nit) capital letter for "ALLOWED_S3_FILE_TYPES" for class attribute is a bit weird for me. I'd prefer allowed_s3_file_types

backported from code in datasets so staying consistent

patrickvonplaten · 2020-11-20T15:36:52Z

src/transformers/hf_api.py

        """
        HuggingFace git-based system, used for models.

        Call HF API to create a whole repo.
+
+        Params:
+            lfsmultipartthresh: Optional: internal param for testing purposes.


can we change lfsmultipartthresh to lfs_multi_part_thresh? => very hard to read :D

i'd rather keep the consistency with the server side on that one

patrickvonplaten · 2020-11-20T15:37:44Z

tests/test_hf_api.py

@@ -73,29 +82,35 @@ def test_whoami(self):

    def test_presign_invalid_org(self):
        with self.assertRaises(HTTPError):
-            _ = self._api.presign(token=self._token, filename="nested/fake_org.txt", organization="fake")
+            _ = self._api.presign(


(nit) no need for _ = => just self._api.presign(...) i good I think

patrickvonplaten

Left a couple of nits. Thanks a lot for the PR! Also cc @agemagician

agemagician · 2020-11-20T15:57:52Z

Thanks a lot for the PR, and thanks for letting me know.

I will give my feedback after testing it with the 11B model soon.

mymusise · 2020-11-21T07:17:37Z

Hi, thanks a lot for the PR.
I try to reinstall transformers with this new branch and follow with this comment:

git clone https://huggingface.co/mymusise/CPM-Third-Party
cd CPM-Third-Party
git lfs install 
transformers-cli lfs-enable-largefiles .

cp ../models/tf_model.h5 ./
git add . && git commit -m 'add model'
git push

Then, after half an hour later, it raised an error:

$ git push
Git LFS: (0 of 1 files) 0 B / 9.68 GB
Git LFS: (0 of 1 files)  4.66 GB / 9.68 GB
EOFoading LFS objects:   0% (0/1), 9.31 B / 9.68 GB
error: failed to push some refs to 'https://huggingface.co/mymusise/CPM-Third-Party'

Did I do something wrong?

julien-c · 2020-11-21T07:58:08Z

@mymusise might have been an intermittent server error. Can you try again?

mymusise · 2020-11-21T08:10:36Z

@mymusise might have been an intermittent server error. Can you try again?

Yes, I try again and again. But it always raises this error at 9.31 GB / 9.68 GB.

👀 But, now git push will return a 502 error :

fatal: unable to access 'https://huggingface.co/mymusise/CPM-Third-Party/': The requested URL returned error: 502

julien-c · 2020-11-21T08:15:27Z

@mymusise Yes, looks like this is crashing/locking the server 😱

Do you mind trying again on Monday? As we'll have more bandwidth to fix then. Sorry about that :/

mymusise · 2020-11-21T08:19:18Z

It's ok, guy, haha. Waiting for your good news.
Happy weekend!

src/transformers/commands/lfs.py

mymusise · 2020-11-24T04:40:47Z

Yes, I try again and again. But it always raises this error at 9.31 GB / 9.68 GB.

Hi, I try again today and push the big model file without any exception! 🎉
Thank guys!

mymusise · 2020-11-24T13:48:27Z

Hi, here I push another model file(4.9GB) again, but this time it gives me a 504 Gateway Time-out error 😓

root@iZt4n9z4x3ph9oc3hhrdneZ:~/CPM-FP16-Third-Party# time git push 
Username for 'https://huggingface.co': mymusise
Password for 'https://mymusise@huggingface.co': 
Counting objects: 3, done.
Compressing objects: 100% (2/2), done.
Writing objects:  66% (2/3), 2.52 GiB | 3.62 MiB/s      
Writing objects: 100% (3/3), 4.46 GiB | 3.77 MiB/s, done.
Total 3 (delta 0), reused 1 (delta 0)
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504 Gateway Time-out
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

real	22m8.653s
user	0m29.758s
sys	0m17.136s

Seems the big file is uploaded completely, it looks like there is some problem with the server configuration about the timeout. @julien-c

Tried three times with the same result.

patrickvonplaten · 2020-11-24T16:51:59Z

@Pierrci - I ran some tests on ~10 GB files (t5-3b) and didn't encounter any problems! However when doing it for ~45GB files (t5-11b), I encounter some problems.

cd t5-11b-repo
transformers-cli lfs-enable-largefiles /path/to/repo
git add .
git commit -m "add"
git push # <= this command failse

In case it is useful, here is a link to the git trace error message: https://github.com/patrickvonplaten/files_to_link_to/blob/master/output.txt

I can always go back to the way of manually uploading to the git-lfs hash path. Maybe the error message is helpful though :-)

Pierrci · 2020-11-24T17:05:24Z

Thanks @patrickvonplaten, I see where it might be coming from, gonna look at it now!

Pierrci · 2020-11-25T00:30:10Z

I deployed a fix that should address your problem, can you try again @patrickvonplaten?

@mymusise Is it possible for you to try again with GIT_CURL_VERBOSE=1 git push so that we can try to get more information? From what you shared so far, your error seems to be different from Patrick's one.

mymusise · 2020-11-25T02:15:12Z

Thanks, @Pierrci. Yes, I think my error is different from Patrick's one. Here is the information with GIT_CURL_VERBOSE=1 git push.
Hope it help.

src/transformers/commands/lfs.py

patrickvonplaten · 2020-11-26T16:32:19Z

I deployed a fix that should address your problem, can you try again @patrickvonplaten?

@mymusise Is it possible for you to try again with GIT_CURL_VERBOSE=1 git push so that we can try to get more information? From what you shared so far, your error seems to be different from Patrick's one.

Awesome it works now @Pierrci - thanks! :-)

Pierrci · 2020-11-30T20:07:06Z

Thanks, @Pierrci. Yes, I think my error is different from Patrick's one. Here is the information with GIT_CURL_VERBOSE=1 git push.
Hope it help.

@mymusise Are you sure LFS is properly installed and configured for the repo? From your logs it seems your git push command isn't doing any LFS work (like running a pre-push hook or calling our LFS endpoint), trying instead to push all the files through the classic git endpoint, which can't work.

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>

initial commit

465ca42

julien-c linked an issue Nov 19, 2020 that may be closed by this pull request

Can't upload the larger model file(9GB) #8662

Closed

julien-c mentioned this pull request Nov 19, 2020

Can't upload the larger model file(9GB) #8662

Closed

[cli] lfs commands

f5462ee

julien-c commented Nov 20, 2020

View reviewed changes

src/transformers/commands/lfs.py Outdated Show resolved Hide resolved

Fix FileSlice

155b4a7

Pierrci reviewed Nov 20, 2020

View reviewed changes

src/transformers/commands/lfs.py Show resolved Hide resolved

julien-c added 21 commits November 20, 2020 11:54

Tweak to FileSlice

d7e32c4

[hf_api] Backport filetype arg from datasets

16e0c62

cc @lhoestq

Silm down the CI while i'm working

b16e6b8

Ok let's try this in CI

754ad6b

Update config.yml

55259e8

Do not try this at home

eaa740e

one more try

c429b7f

Update lfs.py

d69391a

Revert "Tweak to FileSlice"

fbd24cb

This reverts commit d7e32c4.

Update test_hf_api.py

39a09ad

Update test_hf_api.py

a72d12d

Update test_hf_api.py

01b815a

CI still green?

73c082d

make CI green again?

567052b

Update test_hf_api.py

687d99e

make CI red again?

8eff092

Update test_hf_api.py

d7effb5

add CI style back

dddc18c

Fix CI?

d850051

oh my

5f238d9

doc + switch back to real staging endpoint

8e2358e

julien-c marked this pull request as ready for review November 20, 2020 15:06

patrickvonplaten reviewed Nov 20, 2020

View reviewed changes

src/transformers/commands/lfs.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 20, 2020

View reviewed changes

src/transformers/commands/lfs.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 20, 2020

View reviewed changes

src/transformers/commands/lfs.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 20, 2020

View reviewed changes

sgugger approved these changes Nov 20, 2020

View reviewed changes

patrickvonplaten reviewed Nov 20, 2020

View reviewed changes

patrickvonplaten approved these changes Nov 20, 2020

View reviewed changes

Pierrci reviewed Nov 23, 2020

View reviewed changes

src/transformers/commands/lfs.py Outdated Show resolved Hide resolved

julien-c commented Nov 25, 2020

View reviewed changes

src/transformers/commands/lfs.py Show resolved Hide resolved

julien-c and others added 2 commits December 7, 2020 22:11

Apply suggestions from code review

385bda1

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>

Fix docblock + f-strings

8eca31b

julien-c merged commit 28fa014 into master Dec 7, 2020

julien-c deleted the lfs-multipart-uploads branch December 7, 2020 21:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

transformers-cli: LFS multipart uploads (> 5GB) #8663

transformers-cli: LFS multipart uploads (> 5GB) #8663

julien-c commented Nov 19, 2020 •

edited

mymusise commented Nov 20, 2020

patrickvonplaten Nov 20, 2020

patrickvonplaten Nov 20, 2020

julien-c Nov 20, 2020

sgugger left a comment •

edited

sgugger Nov 20, 2020

sgugger Nov 20, 2020

julien-c Nov 23, 2020

sgugger Nov 20, 2020

sgugger Nov 20, 2020

sgugger Nov 20, 2020

sgugger Nov 20, 2020

patrickvonplaten Nov 20, 2020

julien-c Nov 20, 2020

patrickvonplaten Nov 20, 2020

julien-c Nov 20, 2020

patrickvonplaten Nov 20, 2020

julien-c Nov 20, 2020

patrickvonplaten left a comment

agemagician commented Nov 20, 2020

mymusise commented Nov 21, 2020

julien-c commented Nov 21, 2020

mymusise commented Nov 21, 2020

julien-c commented Nov 21, 2020

mymusise commented Nov 21, 2020

mymusise commented Nov 24, 2020

mymusise commented Nov 24, 2020 •

edited

patrickvonplaten commented Nov 24, 2020

Pierrci commented Nov 24, 2020

Pierrci commented Nov 25, 2020

mymusise commented Nov 25, 2020

patrickvonplaten commented Nov 26, 2020

Pierrci commented Nov 30, 2020



		class LfsCommands(BaseTransformersCLICommand):
		# Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs.

transformers-cli: LFS multipart uploads (> 5GB) #8663

transformers-cli: LFS multipart uploads (> 5GB) #8663

Conversation

julien-c commented Nov 19, 2020 • edited

Implementation of a custom transfer agent for the transfer type "multipart" for git-lfs. This lets users upload large files >5GB 🔥.

Things to experiment with:

mymusise commented Nov 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgugger left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickvonplaten left a comment

Choose a reason for hiding this comment

agemagician commented Nov 20, 2020

mymusise commented Nov 21, 2020

julien-c commented Nov 21, 2020

mymusise commented Nov 21, 2020

julien-c commented Nov 21, 2020

mymusise commented Nov 21, 2020

mymusise commented Nov 24, 2020

mymusise commented Nov 24, 2020 • edited

patrickvonplaten commented Nov 24, 2020

Pierrci commented Nov 24, 2020

Pierrci commented Nov 25, 2020

mymusise commented Nov 25, 2020

patrickvonplaten commented Nov 26, 2020

Pierrci commented Nov 30, 2020

julien-c commented Nov 19, 2020 •

edited

sgugger left a comment •

edited

mymusise commented Nov 24, 2020 •

edited