Skip to content

Commit

Permalink
Render chat template tojson filter as unicode (huggingface#31041)
Browse files Browse the repository at this point in the history
* Render chat template tojson filter as unicode

* ruff--
  • Loading branch information
CISC authored May 28, 2024
1 parent 4f98b14 commit 22dab24
Showing 1 changed file with 1 addition and 0 deletions.
1 change: 1 addition & 0 deletions src/transformers/tokenization_utils_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -1853,6 +1853,7 @@ def raise_exception(message):
raise TemplateError(message)

jinja_env = ImmutableSandboxedEnvironment(trim_blocks=True, lstrip_blocks=True)
jinja_env.policies["json.dumps_kwargs"]["ensure_ascii"] = False
jinja_env.globals["raise_exception"] = raise_exception
return jinja_env.from_string(chat_template)

Expand Down

0 comments on commit 22dab24

Please sign in to comment.