fix: round-trip non-UTF-8 text files instead of crashing with UnicodeDecodeError

[vitali87]

Summary

Git diffs any NUL-free file as text, so a latin-1 .txt / .properties / legacy source reached extract_diff and _get_base_file_content, whose strict .decode("utf-8") raised an uncaught UnicodeDecodeError — a raw traceback from pr-split split.

The fix keeps such files' bytes intact end to end:

• extract_diff / _get_base_file_content decode with errors="surrogateescape", and the worker writes files back with the same handler, so the materialised sub-PR file is byte-identical to the dev blob
• the prompt text handed to the Anthropic/OpenAI SDKs goes through _utf8_safe at the single choke point (_count_tokens / _call_llm), replacing undecodable bytes with U+FFFD so the JSON request bodies stay valid
• save_plan writes json.dumps(..., ensure_ascii=True) (which escapes lone surrogates as \udcXX) and load_plan reads via json.loads, so --dry-run → execute round-trips raw_diff losslessly

Stacked on #113 (top of stack #81, where the bytes-based capture from #70 lives).

Test plan

• latin-1 file: extract → parse → materialize → write is byte-equal to the dev blob; _call_llm/_count_tokens receive replaced text; plan save/load round-trips a surrogate-bearing raw_diff — all four fail on the base
• Review verified request bodies for both providers decode as strict UTF-8 and that non-ASCII UTF-8 plans still read back unchanged
• 483 tests pass, ruff clean
• Local review: 5/5 (after one fix→re-review round)

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: af55c3f6-9d7b-4d80-9713-cf993b12eace

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}