-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
No output when using Baichuan2-7B-Chat with 2k input and int4 on XPU #10310
Comments
@NovTi Let's find a better prompt template for Baichuan2. |
@plusbang Take a look. Seems the official template may not working well for Chinese input as well. |
And also <reserved_106> may seem not that user friendly in the raw input string, but seems no better solutions so far? |
I feel we could use the official template, and add some comments to include the reference links and explanatory notes which demonstrates that other template may need for English prompt. |
Updated the prompt format: #10334 |
Issue from @Fred-cell
When using Baichuan2-7B-Chat on XPU device with below question, int4, fp8, fp8_e5m2 and fp32 quantization all generated no output
bigdl-llm
version: 2.5.0b20240303transformers
version: 4.31.0pytorch
version: 2.1The text was updated successfully, but these errors were encountered: