crash when out token length is > 64

[Edward-Lin]

I used to test llama3, when I set max_new_tokens > 64 on windows Ultra Core platform, U9 185 32GB, the program would crash.
attached the code, and logs.
Uploading log_crash.txt…
Uploading run_generation_gpu_woq_for_llama.py.txt…

[Edward-Lin]

run_generation_gpu_woq_for_llama.py.txt
log_crash.txt

[huiyan2021]

Hi @Edward-Lin,

Thanks for reporting, I will try to reproduce first. Could you also help collect the information by running https://github.com/intel/intel-extension-for-pytorch/blob/main/scripts/collect_env.py and upload the result here? Thanks!

btw, are you referring to the guide here: https://intel.github.io/intel-extension-for-pytorch/llm/llama3/xpu/

[Edward-Lin]

ipex-collect-env.txt
please refer the attached FYI.
Yes, I've followed the guide, from https://intel.github.io/intel-extension-for-pytorch/llm/llama3/xpu/, and can run 32 output token, but for the larger, will crash.
BTW, the 2.1.1 chapter showed how to convert model, with different branch, but there is no space to show if we don't want to convert mode, just like to run the mode, which code, or which branch should be used. it's very confused.
Thanks,

[zhuyuhua-v]

ipex-collect-env.txt please refer the attached FYI. Yes, I've followed the guide, from https://intel.github.io/intel-extension-for-pytorch/llm/llama3/xpu/, and can run 32 output token, but for the larger, will crash. BTW, the 2.1.1 chapter showed how to convert model, with different branch, but there is no space to show if we don't want to convert mode, just like to run the mode, which code, or which branch should be used. it's very confused. Thanks,

1. We will further investigate why long sentences are causing crashes.
2. The prolonged conversion time is due to the algorithm used by the quantized model, which requires a long time for iterative generation of accurate quantized data. We are further enhancing support to enable the WoQ LLM solution in ipex to directly handle pre-quantized models provided by the community, simplifying the model quantization process.