upgrade the tgi to V2.0.4

[yuanwu2017]

What does this PR do?

1. Update the tgi to v2.0.4
2. Enable the llava-next
   commad:

volume=$PWD/data
model=llava-hf/llava-v1.6-mistral-7b-hf
docker run -it -p 8083:80 -v $volume:/data --runtime=habana --name tgi-yuanwu -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e SKIP_TOKENIZER_IN_TGI=true -e HABANA_VISIBLE_DEVICES=6,7  -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host tgi:llava --model-id $model --max-input-tokens 4096 --max-total-tokens 8192

client:
from huggingface_hub import InferenceClient
import base64
import requests
import io

client = InferenceClient("http://127.0.0.1:8083")

# read image from local file
image_path = "rabbit.png"
#image_path = "llava_v1_5_radar.jpg"
with open(image_path, "rb") as f:
    image = base64.b64encode(f.read()).decode("utf-8")

image = f"data:image/png;base64,{image}"
prompt = f"![]({image})What is this a picture of?\n\n"

tokens = ''
for token in client.text_generation(prompt, max_new_tokens=100,  stream=True):
    tokens += token
print(tokens)

output:

This is a picture of a rabbit in a space suit. </s>

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@regisss @mandy-li
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[yao-matrix]

@regisss , this is a PR to align tgi-gaudi fork to upstreaming 2.0.4 version pre request, pls help review, thx.

[yuanwu2017]

Test:
commad:

model=meta-llama/Llama-2-7b-hf
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run -p 8083:80 -v $volume:/data --runtime=habana -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host  tgi:llava --model-id $model --max-input-tokens 1024 --max-total-tokens 2048

commad:

model=meta-llama/Llama-2-70b-hf
volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run

docker run -p 8080:80 -v $volume:/data --runtime=habana -e PT_HPU_ENABLE_LAZY_COLLECTIVES=true -e HABANA_VISIBLE_DEVICES=all -e OMPI_MCA_btl_vader_single_copy_mechanism=none --cap-add=sys_nice --ipc=host tgi:llava  --model-id $model --sharded true --num-shard 8 --max-input-tokens 1024 --max-total-tokens 2048

result:

[yuanwu2017]

@regisss Please help to review.

[yao-matrix]

@regisss , aligning to v2.0.4 and llava-next support is required by SynapseAI 1.17 release, could you help review and comment? Thx very much.

[tthakkal]

@regisss @yuanwu2017 Tested Llava models and works as expected,
We should plan to enable Fused_SDPA(USE_FLASH_ATTENTION) here for 1.17. its already enabled and part of optimum-habana main modeling_llava_next.py
Created PR to update documentation in OH here for 1.17 huggingface/optimum-habana#1169

[yuanwu2017]

@tthakkal Thanks. I will have a try. Do you know how to download the maxabs_measure.json file?

[tthakkal]

@tthakkal Thanks. I will have a try. Do you know how to download the maxabs_measure.json file?

@yuanwu2017 Quantization config files are part of OH repo here https://github.com/huggingface/optimum-habana/tree/main/examples/image-to-text/quantization_config

You can start with bf16 with use_flash_attention first and then use fp8(QUANT_CONFIG) as second step

Command below to use flash attention with bf16 in OH

python3 run_pipeline.py \
    --model_name_or_path llava-hf/llava-v1.6-mistral-7b-hf\
    --image_path "https://llava-vl.github.io/static/images/view.jpg" \
    --use_hpu_graphs \
    --bf16 \
    --use_flash_attention

[yuanwu2017]

@tthakkal I will not enable the flash_attention in this PR, because the FW and driver have not been released. After the 1.17 release, I will first upgrade the dockerfile to support 1.17, and then enable falsh_attetion feature. Thanks.

[tthakkal]

@tthakkal I will not enable the flash_attention in this PR, because the FW and driver have not been released. After the 1.17 release, I will first upgrade the dockerfile to support 1.17, and then enable falsh_attetion feature. Thanks.

@yuanwu2017 1.17 is released vault.habana.ai/gaudi-docker/1.17.0/ubuntu22.04/habanalabs/pytorch-installer-2.3.1:latest
OH created branch for its release with 1.17 https://github.com/huggingface/optimum-habana/tree/v1.13-release

[yao-matrix]

@tthakkal I will not enable the flash_attention in this PR, because the FW and driver have not been released. After the 1.17 release, I will first upgrade the dockerfile to support 1.17, and then enable falsh_attetion feature. Thanks.

@yuanwu2017 1.17 is released vault.habana.ai/gaudi-docker/1.17.0/ubuntu22.04/habanalabs/pytorch-installer-2.3.1:latest OH created branch for its release with 1.17 https://github.com/huggingface/optimum-habana/tree/v1.13-release

@tthakkal , our plan is 3 PRs to secure SW stableness and manage regression:

1. align upstreaming 2.0.4 and enable llava-next -> this is already a large PR, we don't want to add more complexities.
2. enable SynapseAI 1.17
3. enable llava-next FP8

@regisss, could you help review and merge this PR, and then we can move forward to step 2 and step 3 quickly, thix.

[regisss]

I just left 3 minor comments

[regisss]

To remove?

[yuanwu2017]

Yes, I want to remove this line. Because it causes the logger cannot print the logs in screen. In latest tgi, there lines are removed. https://github.com/huggingface/text-generation-inference/blob/8dcc7d3f6ba5dacef1405a2cd1bb8e7b7531070f/server/text_generation_server/server.py#L207

[regisss]

Yeah, my point was more about deleting this line for good, but it's a nit anyway 😁

[yuanwu2017]

No problem. When I upgrade the tgi-gaudi to v2.2.0, I will remove these lines.