Is there a way to supply a token to the hugging face inference server run time?

[empath-nirvana]

/kind feature

Describe the solution you'd like
I'd like to be able to provide a token so i can download private hugging face models

I figured out how to add a secret but there's nothing in the code that actually checks environment variables for tokens or will use tokens.

the AutoConfig from_pretrained_model function seems to support a token kwarg, so it should be pretty simple to add.

[yuzisun]

Cc @andyi2it

[andyi2it]

@empath-nirvana You should be able to do that using env variables.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-llama2
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      args:
      - --model_name=<model_name>
      - --model_id=<private_model>
      - --backend=huggingface
      - --task=text_generation
      env:
      - name: HF_TOKEN
        value: <token> or # (envFromSecret)