fix: read Windows-authored agent .md + credentials.json as UTF-8

[Brian Krabach (bkrabach)]

Summary

Three text-decoding gaps that bite files authored on Windows. Each one fails silently or misleadingly rather than raising a clean error. All three are no-ops on POSIX and for any BOM-free file — utf-8 and utf-8-sig decode identically when there is no BOM.

Windows tooling — Notepad, Windows PowerShell 5.1 Set-Content and Out-File — writes UTF-8 with a byte-order mark by default. Reading such a file as plain utf-8 preserves the mark as a leading U+FEFF, which is invisible in every editor but sits at position 0 and defeats any check anchored at the start of the text.

1. A Windows-authored agent .md silently spawns with no tools. spawn.hydrate_agent_overlay read the file as encoding="utf-8". The surviving U+FEFF makes text.startswith("---") False, so the entire file is treated as a plain instruction — tools, hooks, model_role, and meta are dropped from the sub-agent overlay with no error and no warning. The sub-agent then runs with the wrong tools, or none. Now reads encoding="utf-8-sig".

2. A BOM-prefixed credentials.json is reported as corrupt when it isn't. auth._load_credentials read the file with a bare read_text(). The leading U+FEFF reached json.loads, which failed with "Unexpected UTF-8 BOM (decode using utf-8-sig)" — surfaced to the user as "not valid JSON". The file is perfectly valid JSON; the message is both wrong and unactionable. Now reads encoding="utf-8-sig", matching hydrate_agent_overlay.

3. A non-UTF-8 credentials.json crashes the CLI with a raw traceback. The bare read_text() also decoded with the platform default — cp1252 on Windows. Bytes that are genuinely not valid UTF-8 (a file written by an older build under a non-UTF-8 locale, or hand-edited in a legacy encoding) raise UnicodeDecodeError, but the surrounding try caught only json.JSONDecodeError. It propagated uncaught and killed auth list / status / set plus every provider credential lookup behind run, models, and serve. Now caught and re-raised as a click.ClickException naming the file and the remediation.

The atomic write is pinned to encoding="utf-8" for symmetry, so we never author a non-UTF-8 credentials file ourselves.

Evidence — teeth both ways

Native Windows (Python 3.14.3), the real hydrate_agent_overlay() over a BOM+CRLF agent .md:

BASELINE: tools=None                model_role=None      meta=None              keys=['instruction']
FIXED:    tools=['tool-filesystem'] model_role=coding    meta={'name':'my-agent'}

Baseline drops all overlay config, silently. The fix recovers it.

Verified behaviourally against the real functions — 13/13 checks, each run against both the unfixed and fixed code so the failure is confirmed to exist before it is confirmed to be gone:

Check	Unfixed	Fixed
BOM agent .md yields tools / model_role / meta	dropped	recovered
BOM agent .md instruction is frontmatter-free	raw YAML leaked into instruction	clean
BOM-free agent .md parses identically	pass	pass (no regression)
BOM + non-ASCII body decodes correctly	mangled	correct
BOM-prefixed credentials.json loads	"not valid JSON"	loads
Invalid-UTF-8 credentials.json	raw UnicodeDecodeError traceback	ClickException with path + auth clear --force hint
Valid JSON / malformed JSON	pass	pass (unchanged)

Fast gate clean: ruff check, ruff format --check, pyright src/ — 0 errors.

Companion fix upstream

The same class of bug exists in amplifier-foundation's two frontmatter parsers, which sit behind agent metadata, mode metadata, and markdown bundle loading. A BOM there silently empties the parsed frontmatter — a mode declaring default_action: block enforces nothing, and a bundle loses its name and version so it does not resolve.

Fixed separately in microsoft/amplifier-foundation#308 (io/frontmatter.py strips a leading BOM; bundle_docs/frontmatter.py reads with utf-8-sig; 14 regression tests). That PR is independent — neither PR blocks the other, and the two touch disjoint code.

Limits

One Windows machine, one Python (3.14.3). This repo's CI is ubuntu-latest only — there is no Windows leg yet. The BOM behaviour itself is platform-independent (a BOM defeats startswith("---") on any OS), so the agent .md fix is verifiable everywhere; the crash-vs-recover path for credentials.json was exercised on the box.