-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
improve chat template processing #549
improve chat template processing #549
Conversation
708ab4f
to
73c44bb
Compare
please do not review, PR is not ready yet |
73c44bb
to
d3e561b
Compare
PR is ready for review |
209cb1f
to
025dfd3
Compare
atten_mask = new_atten_mask; | ||
} else { | ||
atten_mask = attention_mask; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
attention mask and position ids are specific for stateful and static model runners, while they are not required for PA-based.
I would move it to model runners / pipelines to avoid complexity in generic LLM pipeline impl.
But maybe it should be done during migration to CB rails
CC @Wovchena
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's no other way for now
// If header containt that typical expression we update template and | ||
// extract system message manually from ChatHistory. | ||
std::string header_with_slice = "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}"; | ||
std::string replacement_string = "{% if false %}{% set placeholder = false %}"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you extend your Jinja2Cpp issue to keep track of missing features
8641b8d
to
f54c1da
Compare
16110ba
to
784f269
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed leftover commented lines
Co-authored-by: Zlobin Vladimir <vladimir.zlobin@intel.com>
7f77f89
to
e4e78ac
Compare
…ng used by another process'
TODO
ticket: CVS-143685