Skip to content

Treat cp1252 C1 bytes as Windows‑1252 and coerce non‑UTF‑8 Lua/parsed text to UTF‑8#3

Closed
MjnMixael wants to merge 1 commit into
masterfrom
codex/investigate-rendering-issue-for-unicode
Closed

Treat cp1252 C1 bytes as Windows‑1252 and coerce non‑UTF‑8 Lua/parsed text to UTF‑8#3
MjnMixael wants to merge 1 commit into
masterfrom
codex/investigate-rendering-issue-for-unicode

Conversation

@MjnMixael

Copy link
Copy Markdown
Owner

Motivation

  • The prior change switched fallback decoding to Windows‑1252 but still only reencoded when guessLatin1Encoding() returned true, which can skip cp1252 C1 bytes (e.g. 0x9C for œ) and leave invalid UTF‑8 sequences that trigger utf8::find_invalid() in the scripting API.
  • The goal is to ensure legacy ANSI / Windows‑1252 input is detected and converted to UTF‑8 so it doesn't reach Lua argument validation or other UTF‑8 checks.

Description

  • Added a helper contains_windows_1252_extension_bytes(const char*, size_t) in code/parse/parselo.cpp to detect bytes in the Windows‑1252 C1 extension range (0x80..0x9F).
  • Updated read_raw_file_text() in code/parse/parselo.cpp to use an isLikelyWindows1252 heuristic (combining guessLatin1Encoding() and the new helper) before attempting reencoding, and reencode via Windows‑1252 → UTF‑8 when appropriate.
  • Changed coerce_to_utf8() in code/parse/parselo.cpp to attempt Windows‑1252 → UTF‑8 conversion for likely Western text and return early on success, preventing fallthrough truncation of convertible cp1252 input.
  • Added UTF‑8 coercion for external .lua files in code/scripting/scripting.cpp so Lua source is converted to UTF‑8 before luaL_loadbuffer is called, avoiding invalid byte sequences reaching ade_args.cpp checks.
  • Minor housekeeping: added #include <cstdint> where needed and preserved/used the previously added Encoding_windows_1252 mapping in code/utils/unicode.*.

Testing

  • Ran git diff --check to validate formatting and found no issues (success).
  • Verified repository status with git status --short to confirm only intended files were changed (success).
  • No automated unit tests were changed or added; validation was limited to static checks and inspection of the modified code paths (success).

Codex Task

@MjnMixael MjnMixael closed this Feb 12, 2026
@MjnMixael
MjnMixael deleted the codex/investigate-rendering-issue-for-unicode branch February 12, 2026 20:50
MjnMixael added a commit that referenced this pull request Jul 8, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 8, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
MjnMixael added a commit that referenced this pull request Jul 13, 2026
* allow models for intel entries

* address feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant