[Bug]: Copilot setup crashes on a non-UTF-8 VS Code settings file

[marcelsafin]

Bug Description

Copilot setup preserves an existing .vscode/settings.json when it cannot parse
the file, but a non-UTF-8 file escapes that preservation path and aborts setup
with a raw UnicodeDecodeError. _merge_vscode_settings catches
json.JSONDecodeError and OSError; Path.read_text(encoding="utf-8") raises
the sibling UnicodeDecodeError first.

Steps to Reproduce

1. Create .vscode/settings.json containing non-UTF-8 bytes such as ff fe.
2. Run Copilot integration setup on current main.
3. Setup aborts while reading the existing settings file.

Expected Behavior

Treat the file like the existing JSONC/invalid-JSON path: preserve it, skip the
merge, and emit the existing manual-settings warning.

Actual Behavior

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Specify CLI Version

0.14.4.dev0 (main)

AI Agent

GitHub Copilot

Operating System

macOS

Python Version

Python 3.13.11

Error Logs

UnicodeDecodeError
'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Additional Context

I'd like to fix this by routing UnicodeDecodeError through the existing
preservation/warning branch and adding a regression test that drives
CopilotIntegration.setup with a non-UTF-8 settings file.

[github-actions]

Bug assessment — copilot-settings-unicode-crash: Valid · severity low

---

Bug Assessment: Copilot setup crashes on a non-UTF-8 VS Code settings file

• Slug: copilot-settings-unicode-crash
• Created: 2026-07-29T00:26:13Z
• Source: issue [Bug]: Copilot setup crashes on a non-UTF-8 VS Code settings file #3831
• Verdict: valid
• Severity: low

Report (summarized)

Reported by @marcelsafin. When .vscode/settings.json exists but contains non-UTF-8 bytes (e.g. a UTF-16 LE BOM ff fe), the Copilot integration setup aborts with an unhandled UnicodeDecodeError. The reporter correctly identifies the root cause: _merge_vscode_settings catches json.JSONDecodeError and OSError but not UnicodeDecodeError, which is raised by Path.read_text(encoding="utf-8") before JSON parsing even begins.

Symptom

Running specify init --integration copilot (or re-running Copilot integration setup) when .vscode/settings.json is non-UTF-8 encoded causes an unhandled UnicodeDecodeError crash instead of gracefully skipping the merge and emitting the manual-settings warning.

Expected: treat the file like a JSONC/unparseable file — preserve it, skip the merge, log the warning.
Actual: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Reproduction

1. Create a project directory and initialize a Copilot integration: specify init myproject --integration copilot
2. Replace .vscode/settings.json with a non-UTF-8 file: printf '\xff\xfe{}' > myproject/.vscode/settings.json
3. Re-run setup (or run specify integration install copilot in the project).
4. Observe the crash with UnicodeDecodeError.

Suspected Code Paths

• src/specify_cli/integrations/copilot/__init__.py:499 — dst.read_text(encoding="utf-8") raises UnicodeDecodeError; the except (json.JSONDecodeError, OSError) clause on line 500 does not catch it, so the exception propagates unhandled.
• src/specify_cli/integrations/copilot/__init__.py:489 — _merge_vscode_settings method; this is the sole location that needs changing.

Root Cause Hypothesis

UnicodeDecodeError is a subclass of ValueError, not of OSError or json.JSONDecodeError. The except clause on line 500 therefore misses it entirely. A non-UTF-8 settings file is a perfectly normal user state (e.g. VS Code itself can write UTF-16 LE files, or a Windows editor may produce them), so this is a legitimate unhandled edge-case. Confidence: high — the code path is a single line, the exception hierarchy is unambiguous, and the reporter provided exact byte values.

Proposed Remediation

Preferred: Add UnicodeDecodeError to the existing except tuple on line 500:

except (json.JSONDecodeError, OSError, UnicodeDecodeError):

This routes the non-UTF-8 path through the already-tested preservation/warning branch with no behaviour change for any other path. The fix is a one-character change that is easy to review and carries no risk of regressions.

Alternatives:

• Use errors="replace" or errors="ignore" in read_text() to tolerate non-UTF-8 bytes and let JSON parsing attempt to recover. Trade-off: more complex, the resulting string is likely still invalid JSON so the JSONDecodeError branch fires anyway — adds complexity for no benefit.
• Open the file in binary mode, attempt UTF-8 decode manually, and fall through to the warning on failure. Trade-off: equivalent outcome, more code.

Files likely to change:

• src/specify_cli/integrations/copilot/__init__.py (line 500 — one-word addition to the except tuple)
• tests/integrations/test_integration_copilot.py (add regression test)

Tests to add or update:

• Add a test that writes a non-UTF-8 bytes settings file to a temp dir, calls CopilotIntegration._merge_vscode_settings(src, dst), and asserts: (a) no exception is raised, (b) the destination file is unchanged, (c) the warning log is emitted. This mirrors the existing JSONC/comment test pattern.

Risks & Considerations

• No API breakage: _merge_vscode_settings is a private static method; no public interface changes.
• No data risk: the fix widens the "preserve and warn" branch, which is already the safe fallback — it cannot destroy user data.
• Blast radius: affects only the Copilot integration's settings merge step; no other integration calls this method.
• Long-standing vs regression: the OSError catch was clearly intended to cover read failures; omitting UnicodeDecodeError is an oversight, not a deliberate choice. This has presumably existed since the method was introduced.

Open Questions

• None — the report is self-contained and the fix is unambiguous.

Generated by 🐛 Assess Bug from Labeled Issue for issue #3831 · 73 AIC · ⌖ 10.9 AIC · ⊞ 34.2K · ◷