Skip to content

[hf-xet v1.8.0]: Direct-to-S3 xorb uploads (experimental)

Latest

Choose a tag to compare

@github-actions github-actions released this 30 Oct 10:00
e551f0a

✨ New Features

  • Experimental direct-to-S3 xorb uploads: xorbs are uploaded straight to S3 through short-lived upload grants issued by CAS, which validates each xorb before storing it. The server is enabling this gradually; other uploads use the existing path (by @assafvayner in #1004 #1005 #1008 #1029)
    • Firewall note: allow s3.us-east-1.amazonaws.com (and s3.eu-west-3.amazonaws.com for repos stored in the EU). If S3 can't be reached, uploads fall back to the existing path.
    • Set HF_XET_CLIENT_LEGACY_DIRECT_XORB_UPLOAD=1 to always use the existing path.

🐛 Bug Fixes

  • Downloads no longer request reconstruction information past the end of the file if file size is known ahead (by @seanses in #995)
  • Fix a partially written file being published after a write error (by @k1nkreet in #1067) - 1st contribution, thanks!

Full Changelog: v1.7.1...v1.8.0