-
Notifications
You must be signed in to change notification settings - Fork 6.4k
purge HF_HUB_ENABLE_HF_TRANSFER; promote Xet #12497
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for starting it
accelerate \ | ||
numpy==1.26.4 \ | ||
hf_transfer \ | ||
hf_xet \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's keep the changes in docker
limited to these ones:
diffusers/.github/workflows/build_docker_images.yml
Lines 73 to 77 in f072c64
- diffusers-pytorch-cpu | |
- diffusers-pytorch-cuda | |
- diffusers-pytorch-xformers-cuda | |
- diffusers-pytorch-minimum-cuda | |
- diffusers-doc-builder |
env: | ||
DIFFUSERS_IS_CI: yes | ||
HF_HUB_ENABLE_HF_TRANSFER: 1 | ||
HF_XET_HIGH_PERFORMANCE: 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will there be any problems stemming from this in case a repo isn't Xet-transferred? I am guessing this is currently the case i.e., not all repos have gone through this, yet. In that case, would using HF_HUB_ENABLE_HF_TRANSFER
have any advantage?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good question, all repos on the hub are powered by xet
. Hence HF_HUB_ENABLE_HF_TRANSFER
has no usage whatsoever there
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks 👍🏽
I think if we address #12497 (comment), the CI won't cry. |
Sorry for not understanding this correctly - but the whole point is to remove all mentions of hf_transfer (it has no effect on xet repos i.e. 100% of the hub) - so I'd rather fix why the CI is not happy with xet in the first place. More so, the CI errors look more related to model specific integration test that hub - let me know if my understanding is incorrect. |
cc @DN6 |
I have pushed a change in 708973b, which should fix the issue. The concerned Docker containers were previously not used in our CI (they are there for bookkeeping and will likely be removed soon). |
Failing tests are unrelated. |
(as the title state - since hf_transfer has no effect w/ xet, let's move to hf_xet)