-
Notifications
You must be signed in to change notification settings - Fork 107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement v2.llama3 grammar sampling + some API request edge cases error handling #189
Conversation
…ng code_interpreter tool in server_vllm
functionary/inference.py
Outdated
)[0] | ||
+ prompt_template.fn_param_sep_token | ||
) | ||
final_prompt += "all" + prompt_template.fn_param_sep_token |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we create another abstract class for get_force_text_response_prefix() --> "" by default
and in template_v2, we will overwrite this by returning: "all\n". So in the future, if we change the template we don't have to if-else at this ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure ok
return ["<|start_header_id|>assistant<|end_header_id|>\n\n"] | ||
|
||
def get_stop_tokens_for_generation(self) -> List[str]: | ||
return ["<|eot_id|>", "<|end_of_text|>"] | ||
|
||
def get_start_of_function_call_token(self) -> str: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can remove this, right ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, not sure why it was here. Removed
@@ -312,3 +553,4 @@ def get_chat_template_jinja(self) -> str: | |||
|
|||
def get_force_function_call_prefix(self, function_name: str): | |||
return f"{self.function_separator}{function_name}\n" | |||
return f"{self.function_separator}{function_name}\n" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh why made it repeated ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it was accidental. Removed
v2 FSM:
v2.llama3 FSM: