Read JSON and write XML as UTF-8 regardless of locale - #655
Open
davidpavlovschi wants to merge 1 commit into
Open
Read JSON and write XML as UTF-8 regardless of locale#655davidpavlovschi wants to merge 1 commit into
davidpavlovschi wants to merge 1 commit into
Conversation
Fixes elapouya#516 docxtpl opened the only two text files it handles without an explicit encoding, so both fell back to locale.getpreferredencoding(False) : - docxtpl/__main__.py : `python -m docxtpl` decoded the json data with the locale code page. Under cp1252 the UTF-8 bytes silently become mojibake in the generated docx, which is what elapouya#516 reports ; under cp936 or a C locale they raise UnicodeDecodeError instead. JSON is UTF-8 (RFC 8259), so the encoding is not a guess. "utf-8-sig" is used rather than "utf-8" to also skip the BOM that PowerShell and Windows editors write. A file that really is not UTF-8 is now reported like any other bad input instead of escaping the command line error handling. - docxtpl/template.py : write_xml() encoded the document xml with the same locale code page and raised UnicodeEncodeError as soon as the document contained a character the code page cannot represent (any CJK or Cyrillic text under cp1252, any non-ASCII text under a C locale). This is also why the bug looked unreproducible : on Linux and macOS the default encoding already is UTF-8. tests/utf8_locale.py re-runs itself in a child interpreter forced off UTF-8 (PYTHONUTF8=0 + C locale) so the regression reproduces on Linux and macOS too. Reverting either fix on its own makes it fail. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
utf-8-siginstead of the host locale, fixing Wrong encoding when runned via module #516 while accepting Windows-style UTF-8 BOMs.UnicodeDecodeErrortraceback.DocxTemplate.write_xml()output as UTF-8 instead of the host locale.The
write_xml()half has no linked issue; I found it while tracing the same locale-dependent text-I/O pattern for #516.Verification
I independently ran the locale regression, all 37 script tests, Flake8, and Black on macOS with Python 3.9.6:
The test forces Python out of UTF-8 mode under a
Clocale. It also verifies that invalid Latin-1 JSON raises the CLI's normalRuntimeError, and thatwrite_xml()emits bytes that decode as UTF-8.Current CI does not run these scripts because
test.ymlinvokestests/runtests.pyfrom the repository root. PR #652 already addresses that runner problem, so this PR does not overlap it.Related: withdrawn PR #640 proposed the CLI half in June and closed without comments or review. This change was developed independently and also covers BOM input, the CLI error path, and
write_xml().AI disclosure
Claude Opus 5 implemented and tested this change. OpenAI Codex independently reviewed the diff and reran the regression, full script suite, and linters. David authorized this automated contribution workflow and owns the submission. The commit keeps an explicit
Co-Authored-Bytrailer for Claude.