[Evaluation] Fix UTF-8 encoding for red team JSONL files on Windows

[slister1001]

What

Add explicit encoding=utf-8 to all open() calls in the PyRIT result processing path for red team scans.

Why

On Windows, Python open() defaults to the system locale encoding (cp1252/charmap). When JSONL files contain non-ASCII characters (from UnicodeConfusable strategy or CJK language prompts), reading them back fails with UnicodeDecodeError. This causes 0 attack_details in final results.

What Changed

4 open() calls fixed across 2 files:

• _result_processor.py:202 - data file read
• _utils/formatting_utils.py:306 - existing file line count read
• _utils/formatting_utils.py:338 - JSONL write (replace path)
• _utils/formatting_utils.py:378 - JSONL write (new file path)

Testing

All 367 red team unit tests pass. Fixes bug bash tests 1.7 (UnicodeConfusable) and 1.16 (Japanese/Chinese).

Risk

None - the write side already produces UTF-8 content; this makes the read side match.

[Copilot]

Pull request overview

This PR addresses Windows-specific UnicodeDecodeError failures when reading red team JSONL files that contain non-ASCII text by ensuring UTF-8 is explicitly used for JSONL file I/O in the PyRIT result processing flow.

Changes:

• Add encoding="utf-8" to JSONL reads in the red team result processor.
• Add encoding="utf-8" to JSONL read/write operations in red team formatting utilities.
• Add unit tests intended to cover Unicode JSONL round-trips; update version/changelog for the next release.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_result_processor.py	Reads JSONL data files using explicit UTF-8 to avoid Windows locale decode errors.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_utils/formatting_utils.py	Uses explicit UTF-8 when reading existing JSONL and writing JSONL output files.
sdk/evaluation/azure-ai-evaluation/tests/unittests/test_redteam/test_formatting_utils.py	Adds tests intended to validate Unicode JSONL read/write behavior.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_version.py	Bumps package version to 1.16.0.
sdk/evaluation/azure-ai-evaluation/CHANGELOG.md	Adds an Unreleased 1.16.0 entry describing the UTF-8 fix.

[nagkumar91]

Core fix is correct — adding encoding="utf-8" to all 4 open() calls is the right approach for Windows compatibility.

Minor suggestions (non-blocking):

• The new tests do manual open() round-trips but don't exercise the actual production functions (write_pyrit_outputs_to_file() / to_red_team_result()). Consider adding integration tests that call those functions with Unicode data to validate the complete fixed path.
• Note: PR #45502 also bumps the version to 1.16.0, so whichever merges second will hit a conflict on _version.py and CHANGELOG.md.