-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Phi-3 medium 128k instruct fails to start #1930
Comments
Have you tried adding "attention_bias": false to the config.json? I used a local volume to save the model and altered the config as described. It works (tested with image ghcr.io/huggingface/text-generation-inference:2.0.3). |
I encounter this as well. I believe it arises from the recent addition of Granite support after Phi-3 support in TGI 2.0.3. See here. |
@OjoDojoJo What's your full command line? I'm running this command on a aws g6.48xlarge
And I'm getting this error:
|
Can confirm that this works. There's currently an open PR on HF to fix the issue. In the meantime, you can run the model by directly specifying the revision. Here's my full command:
|
I'm still getting the same issue as @amihalik, even with the attention bias fixed:
Not sure what causes it, I'm using pretty much the exact same docker commands. |
Still fails for me with TGI 2.0, trust remote code, attention_bias false.
|
It is the same for us. tells me |
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
System Info
Information
Tasks
Reproduction
Expected behavior
The text was updated successfully, but these errors were encountered: