Enable dtype support for huggingface server

[Datta0]

What this PR does / why we need it: Currently, HF runtime doesn't have support to switch dtypes while VLLM does. And VLLM seems to load 16bit by default and HF seems to be running at 32bit causing inconsistencies.

Type of changes

• New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing:

Please describe the tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

• Enable --dtype=float16 (or bfloat16 or float32) argument while running huggingfaceserver via python -m huggingfaceserver --model_id=meta-llama/Llama-2-7b-chat-hf --model_name=llamachat --backend=huggingface. Check for dtype of weights and memory usage.

[yuzisun]

Typically BF16 is for training, for inference FP16 is more acceptable

[yuzisun]

is this the argument defined from vllm ?

[Datta0]

Yes it is defined here. I verified the dtype of the model after engine creation and it matches with the input.
Memory usage matches up too (before initialising cache and blocking memory for blocks of paged attention)

[yuzisun]

Can we add a comment that this argument is used by both HF and vllm?

[yuzisun]

also the default should be auto ?

[Datta0]

auto defaults to FP16 even if the model is FP32 in case of vLLM
but for HF, it doesn't. It sticks to FP32. And HF config doesn't have the info about what type the model weights are of. So for FP32 models (which are rare but do exist. GPTJ, OLMo etc for eg), this would again cause inconsistency.

[yuzisun]

@Datta0 Can you help sign off your commit?

[Datta0]

@yuzisun Done signed off the commits :)

[Datta0]

@yuzisun can you please check this :)

[yuzisun]

@yuzisun can you please check this :)

@Datta0 can you help rebase the PR to resolve the conflicts?

[datta-nimmaturi]

@yuzisun Done :)

[yuzisun]

@Datta0 the e2e test failed here

  =====================================  Logs for Predictor Pod: hf-t5-small-predictor-00001-deployment-c54b8fc6-tbcmp  =========================================
  INFO:kserve:Loading generative model for task 'text2text_generation'
  ERROR:kserve:Failed to start model server: HuggingfaceGenerativeModel.__init__() got an unexpected keyword argument 'dtype'
  ================================================================================================================

[yuzisun]

I think the merge conflict is not resolved correctly

[yuzisun]

this file should no longer exist

[datta-nimmaturi]

Ooopsie...
Updated it now

[yuzisun]

This logic seems like can be folded into later on line 147

dtype = kwargs.get("dtype", "auto")
hf_dtype_map = {
             "float32": torch.float32,
             "float16": torch.float16,
             "bfloat16": torch.bfloat16,
             "half": torch.float16,
             "auto": torch.float16
             "float": torch.float32,
         }

[datta-nimmaturi]

auto defaults to FP16 even if the model is FP32 in case of vLLM. So if we do the mapping only for HF, for an FP32 model, HF loads in 32 and vLLM would load in FP16

[yuzisun]

But here auto maps to FP16 if dtype is not provided, did I miss anything?

[Datta0]

Yeah so if the base model itself is BF16, vLLM would load in BF16. HF (with this mapping) would load in FP16. Memory wise both are pretty much the same. But it still is an inconsistency right?

[yuzisun]

I am not sure why vLLM defaults to BF16 which only makes sense for training for convergence, but for inference we should use FP16 which performs more calculations per unit time.

[yuzisun]

oh I think I get your point now, you are trying to set default to FP16 for vLLM

[yuzisun]

Based on HF doc https://huggingface.co/docs/transformers/en/main_classes/model, user might actually expect loading the model using the dtype from config.json.

"auto" - A torch_dtype entry in the config.json file of the model will be attempted to be used. 

[Datta0]

Yes...
But auto is BF16 is only for BF16 models aka weight tensors are themselves stored in BF16 like llama3
Yeah there's compelling reason to default to FP16 (at least for inference to support older architectures)
IDK why vLLM doesn't do that
and the conversion is straight forward anyway

[Datta0]

Should we then allow only FP16, BF16 and FP32 as allowed dtypes for both HF and vLLM? That will avoid the confusion.

[yuzisun]

I think we still want auto, it is fine as long as we clearly document the behavior.

[yuzisun]

For encoder models we probably do not want to default to FP16, see

2024-05-03 18:13:23.626 1 kserve ERROR [inference_error_handler():104] Exception:
  Traceback (most recent call last):
    File "/huggingfaceserver/huggingfaceserver/encoder_model.py", line 242, in predict
      outputs = self._model(**input_batch).logits
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
      return forward_call(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward
      outputs = self.bert(
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
      return forward_call(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward
      embedding_output = self.embeddings(
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
      return forward_call(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 239, in forward
      embeddings = self.LayerNorm(embeddings)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
      return self._call_impl(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
      return forward_call(*args, **kwargs)
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/modules/normalization.py", line 196, in forward
      return F.layer_norm(
    File "/prod_venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2543, in layer_norm
      return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
  RuntimeError: "LayerNormKernelImpl" not implemented for 'Half'

[yuzisun]

Actually a lot of CPU-based operations in Pytorch are not implemented to support FP16, so we probably need to check need GPU availability.

[Datta0]

Thats a fair point. But people would probably want to run models at lower precisions on CPU given that there's not much to lose. We should at least support that right? And for those who want to run, its for themselves to verify compatibility/support?

[Datta0]

so should we default to float32/float16 based on GPU availability?
something like

default_dtype = 'auto' if torch.cuda.is_available() else 'float32'
dtype = kwargs.get("dtype", default_dtype)

Edit: Ok I just checked vLLM CPU support document. For CPU vLLM casts FP16 to BF16...
https://arc.net/l/quote/nujjbyig

[spolti]

/lgtm

[yuzisun]

/lgtm
/approve

[oss-prow-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Datta0, spolti, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment