Support for private models from huggingface.co

[julien-c]

Add a use_auth_token flag (or string) to all from_pretrained entry points, to specify token to use as Bearer authorization for remote files.

• if it's a string, use it
• If it's true, will get token from ~/.huggingface/token (will throw if no token there)

You can test this with:

model = AutoModelForMaskedLM.from_pretrained("pierric/hf-private", use_auth_token=True)

We'll add unit tests down the line but need to think about which environment those tests are going to hit.

⚠️ For now, I decided against adding token by default to all calls if user is logged in. Let's discuss though!

[sgugger]

Looking good, though the documentation needs tweaking IMO.

[sgugger]

Suggested change

	Specify token to use as Bearer authorization for remote files. If true, will get token from
	~/.huggingface.
	The token to use as bearer authorization for the remote files. If :obj:`True`, will use the token
	generated when running :obj:`transformers-cli login` (stored in :obj:`~/.huggingface`).

Think it's better to tell the user what to do to get that token in the right place than where the token is stored (unless I misunderstood where that token is coming from).

This might also warrant a note like:

            .. note::

                Passing a token is required when you want to use a private model.

if the default is to not use the token when a user is logged in.

[julien-c]

makes sense @sgugger. I'll wait for other potential comments and batch them tomorrow.

Re. the note, where would I add it? e.g. https://huggingface.co/transformers/philosophy.html#main-concepts ?

Thoughts on your side on the "should make it enabled by default" question?

[sgugger]

I was thinking in the doc string of from_pretrained, but it can also go there. The more the merrier since people don't read all the doc.

[sgugger]

As for "should make it enabled by default", I have no strong feeling either way. It makes sense to me to have some added code where something won't work out of the box for all users. It also makes sense to want logged-in users to be able to use it automatically. I think we can start with your approach and see users thoughts.

[Pierrci]

⚠️ For now, I decided against adding token by default to all calls if user is logged in. Let's discuss though!

I think that for a more seamless experience it makes sense to add the token automatically for logged-in users, but otherwise LGTM!

[patrickvonplaten]

I think I'd vote for use_auth_token to be just Optional[str] and to automatically allow access to privet model by default if user is authorized. I don't see a reason why someone would not want to have access to his/her model if authorized.

IMO, this is both cleaner in terms of design (don't like Union[str, bool]) and a nicer UX.

[Narsil]

LGTM.

On the default approach, I am also in favor of taking the token by default too.
The only drawback I can see is that now code depends on the token file, which might lead to some confusion for people running code in a different place. We probably should make sure the error they receive is something different than 404 does not exist but 403 forbidden so users can figure out what is the problem.

What do you think ?

[julien-c]

@patrickvonplaten I don't follow here. In this PR we want to have a way to either pass a token directly, or to opt in to use the one that's store in ~/. Don't see how I can do that with just an optional string?

[patrickvonplaten]

@patrickvonplaten I don't follow here. In this PR we want to have a way to either pass a token directly, or to opt in to use the one that's store in ~/. Don't see how I can do that with just an optional string?

I might have misunderstood a bit what constrains there are on the functionality. I thought, the following logic is possible and makes sense here:

• If user passes a string use_auth_token, then use this as the token
• Else look for token in ~/.huggingface:
  - if there is no token and model is private -> throw error
  - if there is no token and model is not private -> load the model as usual
  - if there is a token -> use this one

Not sure if there is something I am completely overlooking here in the logic though, e.g. if we cannot know before hand whether the model is private or not

[patrickvonplaten]

@patrickvonplaten I don't follow here. In this PR we want to have a way to either pass a token directly, or to opt in to use the one that's store in ~/. Don't see how I can do that with just an optional string?

I might have misunderstood a bit what constrains there are on the functionality. I thought, the following logic is possible and makes sense here:

• If user passes a string use_auth_token, then use this as the token
• Else look for token in ~/.huggingface:
  • if there is no token and model is private -> throw error
  • if there is no token and model is not private -> load the model as usual
  • if there is a token -> use this one

Not sure if there is something I am completely overlooking here in the logic though, e.g. if we cannot know before hand whether the model is private or not

Ok never mind - as discussed offline this would require more features to add which is out-of-scope for this PR -> so LGTM!

[julien-c]

Also cc'ing @borisdayma as this PR adds a exist_ok param to HfApi.create_repo()

[LysandreJik]

Cool, very nice functionality. I like that it doesn't log anything more than the user agent in the default case.

LGTM