Xet upload workflow

[hanouticelina]

Partially resolves #2713

This PR contains only the Xet upload workflow implemented in xetpoc_huggingface_hub (internal).

The main branch for xet storage integration is xet-integration.

Main changes:

• Make hf_xet available as an optional dependency via pip install huggingface_hub[hf_xet]
  Note: since it's a common part for download and upload, this has been pushed directly into the xet-integration branch.
• Integrate changes from the xet poc for the download workflow only.
• Add tests.

to try it in from this branch:

pip install -e ".[dev,hf_xet]"
export HF_DEBUG=1 #  if you want to set huggingface_hub logger to debug level
huggingface-cli upload --repo-type model rajatarya/xet-enable-test ~/data/yelp_sample

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[bpronan]

There are a few missing doc strings, but otherwise looks great!

[Wauplin]

Really nice! 🔥

Implementation looks good to me. I just have a question about when Xet is enabled but user is not xet authorized. Wondering when could that happen and what the expected behavior here (my understanding is that currently it fails to upload without any workaround possible).

[Wauplin]

Suggested change

	**How it works:**
	**How it works:**

[Wauplin]

I feel that the whole section would be better suited for https://github.com/huggingface/huggingface_hub/pull/2899/files#diff-b1fa48a1bcb46429d1a7d311268539b7fe814e4158608cce9c14d5d60a9c4a51 i.e. in official docs rather than the docstring. And keep things simple in the docstring with a "check out this guide to learn more about how it works".

Can be done in a later PR or in the docs PR directly (let's not block this one just for documentation)

[hanouticelina]

yes, actually i put it there mainly for users who will read the code since we have a "blackbox" call to hf_xet.upload_files (same for downloading). but yes I agree it's better to have it in a documentation and then link it in the docstring. will do that in a following PR 👍

[hanouticelina]

I just have a question about when Xet is enabled but user is not xet authorized. Wondering when could that happen and what the expected behavior here (my understanding is that currently it fails to upload without any workaround possible).

I think if xet is enabled in the repo and hf_xet is installed, then we should not fallback to http since it means that the user passed invalid credentials when calling GET /api/:repoType(models|spaces|datasets)/:namespace?/:repo/xet-write-token/:rev.
it's actually the same auth level as upload, delete and commit (see here: moon-landing/server/server.ts#L834C1-L835C1).

[Wauplin]

I think if xet is enabled in the repo and hf_xet is installed, then we should not fallback to http since it means that the user passed invalid credentials when calling GET /api/:repoType(models|spaces|datasets)/:namespace?/:repo/xet-write-token/:rev.
it's actually the same auth level as upload, delete and commit (see here:moon-landingserver/server.ts#L834C1-L835C1).

Ah yes, I forgot the workaround in this case would be to uninstall hf_xet if really they are not authorized to use it. But shouldn't happen except if doing something unexpected. Then it's all fine to raise the unauthorized error :)

[Wauplin]

🔥

[hanouticelina]

Let's wait for a final review from @bpronan and/or @rajatarya and then I think we should be good to merge! 🙂

[bpronan]

[bpronan]

I think if xet is enabled in the repo and hf_xet is installed, then we should not fallback to http since it means that the user passed invalid credentials when calling GET /api/:repoType(models|spaces|datasets)/:namespace?/:repo/xet-write-token/:rev.
it's actually the same auth level as upload, delete and commit (see here: moon-landing/server/server.ts#L834C1-L835C1).

This is 💯 correct. The user would only see this case (xetEnabled on repo + hf_xet installed + no access) if they don't have upload/commit access on the repo in the first place. In other words, they have to have write access to the repo to upload to a xet enabled repo with hf_xet installed. The earlier call to _fetch_upload_modes would have failed already by then.

Uninstalling hf_xet shouldn't help here because they can't write to the repo anyway.

[hanouticelina]

I think we can merge this one to the xet-integration branch and then we can open another PR for any xet-specific part once we merge the If-None-Match path into main.