Enable bare messages= input on Relevance/Similarity/Fluency/Retrieval/ResponseCompleteness + all RAI evaluators

[Rabah B (RabsB)]

Summary

Enable bare messages=[...] kwarg input (plus optional scalar context, ground_truth, tool_definitions adjuncts) on the evaluators most commonly targeted by cloud batch-execution engines whose data_mapping produces this shape. This is a plumbing fix, not a semantics change — every affected evaluator was already capable of scoring the input via either the scalar (query, response, ...) path or the conversation={"messages": [...]} path. The only gap was that no code path unpacked the specific messages=<list> + top-level scalar-adjunct call pattern into either of those existing forms.

Problem

When a customer's evaluation data_mapping targets a messages field (plus optional adjuncts):

"data_mapping": {
    "messages":         "{{item.messages}}",
    "context":          "{{item.context}}",
    "ground_truth":     "{{item.ground_truth}}",
    "tool_definitions": "{{item.tool_definitions}}"
}

the SDK batch engine invokes each evaluator as:

evaluator(messages=[...], context=..., ground_truth=..., tool_definitions=[...])

Evaluators with only (query, response, ...) and (conversation) overloads then failed either with:

• EvaluationException: Either 'conversation' or individual inputs must be provided. — the base class singleton matcher found no overload with a messages param, so nothing was routed to _do_eval.
• EvaluationException: Cannot provide both 'conversation' and individual inputs at the same time. — if the evaluator had a custom _convert_kwargs_to_eval_input that special-cased conversation, the leftover scalar context/ground_truth/tool_definitions at the top level tripped the ambiguity check.

Neither failure mode reflected an actual limitation of the evaluator; both were routing gaps.

Fix

New shared helper: azure.ai.evaluation._evaluators._common.hoist_messages_to_conversation

Normalises a bare messages=[...] kwarg (plus any scalar adjuncts alongside it) into conversation={"messages": [...], ...} so EvaluatorBase._derive_conversation_converter — the existing per-turn q/r/context/ground_truth extractor — is what runs. A top-level scalar ground_truth is stamped onto each assistant turn that does not already carry a per-turn ground_truth so the per-response extraction path picks it up. Zero effect when conversation is already provided or when messages is absent.

Applied to 5 evaluators directly (PromptyEvaluatorBase subclasses with a conversation overload):

• RelevanceEvaluator — validator swap + convert override
• SimilarityEvaluator — validator init + reordered convert override (retains pre-existing q/r-required checks when neither conversation nor messages is provided)
• FluencyEvaluator — validator swap + convert override
• RetrievalEvaluator — convert override (no validator to swap)
• ResponseCompletenessEvaluator — convert override (no validator to swap)

Applied once in RaiServiceEvaluatorBase._convert_kwargs_to_eval_input, inheriting to all 8 RAI safety evaluators in one change:

• ViolenceEvaluator
• HateUnfairnessEvaluator
• SelfHarmEvaluator
• SexualEvaluator
• ProtectedMaterialEvaluator
• IndirectAttackEvaluator (XPIA)
• CodeVulnerabilityEvaluator
• ECIEvaluator

The RAI base already routes conversation into _evaluate_conversation; the hoist just lets messages= reach that existing path instead of failing kwarg matching. No RAI service backend change required.

Total: 13 evaluators fixed across 4 focused commits.

Verified end-to-end

Live gpt-4o-mini scoring against tiger-5.openai.azure.com for every affected input shape on the direct-file-edit evaluators (RAI verified at kwarg-processing layer since RAI service calls require an Azure AI project scope):

• Relevance / Similarity / Fluency / Retrieval / ResponseCompleteness with messages=[...] — score returned, no errors.
• Same with messages=[...] + context= + ground_truth= + tool_definitions= — score returned, no errors.
• Same evaluators with legacy scalar (query, response, ground_truth) — score returned unchanged. Regression clean.
• Similarity with query=None after fix — still rejected with the exact pre-existing error message. Safety net intact.

Explicitly not in scope

Evaluators without a conversation overload — IntentResolutionEvaluator, the metric family (F1ScoreEvaluator, BleuScoreEvaluator, RougeScoreEvaluator, MeteorScoreEvaluator, GleuScoreEvaluator), and the tool family (ToolCallAccuracyEvaluator, ToolInputAccuracyEvaluator, ToolOutputUtilizationEvaluator, ToolCallSuccessEvaluator) — would need a different helper (project last assistant turn → scalar response) plus per-evaluator semantic decisions on how a message list splits into query/response/tool_calls. All 10 of these are also declared as supported_evaluation_levels: [turn] in the customer-facing evaluator catalog, so they self-skip at conversation level today and the customer-visible impact of extending them to bare messages= is bounded. Left as follow-up.

Files changed

• new sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_conversation_input.py
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/__init__.py — export the new helper
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py — inherit hoist across all RAI evaluators
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_retrieval/_retrieval.py
• sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_response_completeness/_response_completeness.py

Testing recommendations for reviewers

• Existing tests/unittests/test_common_validators.py::TestMessagesOrQueryResponseInputValidator continues to cover the validator swap.
• The hoist_messages_to_conversation helper is small enough that a targeted unit test file exercising the three transform variants (no conversation, existing conversation, bare messages + adjuncts) would give straight-line coverage. Left for a follow-up per repository-team preference on test placement.
• E2E model-integration tests were run locally against a real Azure OpenAI deployment; results summarized in commit messages.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Enables evaluators to accept a bare messages=[...] kwarg (plus optional context, ground_truth, tool_definitions) by normalizing it into the existing conversation={...} input shape, fixing batch-engine routing failures without changing evaluation semantics.

Changes:

• Added shared hoist_messages_to_conversation helper to convert messages-shaped kwargs into conversation kwargs.
• Applied the helper to multiple Prompty-based evaluators and the RAI service evaluator base to unblock messages= input across the targeted evaluator set.
• Tightened the openai dependency spec with an upper bound.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
sdk/evaluation/azure-ai-evaluation/setup.py	Adds an upper bound to the openai dependency range.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_conversation_input.py	Introduces shared normalization helper to hoist messages into conversation.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/init.py	Exports the new helper from the common evaluator package.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py	Uses the helper so all RAI evaluators accept messages= input.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py	Switches validator + hoists messages= into conversation before base conversion.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py	Adds validator and hoists messages= while preserving single-turn q/r validation behavior.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py	Switches validator + hoists messages= into conversation before base conversion.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_retrieval/_retrieval.py	Hoists messages= into conversation before base conversion.
sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_response_completeness/_response_completeness.py	Hoists messages= into conversation before base conversion.

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (4)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• The replacement validator's messages branch checks only dictionary shape and roles, unlike the previous ConversationValidator, which also requires and validates each message's content. For example, [{'role': 'user'}, {'role': 'assistant'}] now passes validation and is converted into empty query/response strings that are sent to the judge. Preserve full conversation validation for the bare-messages path, either by hoisting before the original validator runs or by strengthening the shared messages validator.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• The replacement validator's messages branch checks only dictionary shape and roles, unlike the previous ConversationValidator, which also requires and validates each message's content. Thus role-only messages pass and are converted into an empty response that is scored. Preserve full conversation validation for the bare-messages path, either by hoisting before the original validator runs or by strengthening the shared messages validator.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• This validator is never invoked by SimilarityEvaluator: neither this class nor PromptyEvaluatorBase calls _validator.validate_eval_input, and the conversion method immediately hoists messages away. Consequently malformed bare-message inputs bypass the validator that this change initializes. Validate the raw messages path before hoisting (without changing the existing scalar error ordering), or remove the unused validator and perform equivalent validation in the converter.

        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/tests/unittests/test_conversation_input.py:24

• These tests exercise only the helper in isolation, so they do not verify that the five evaluator overrides and the RAI base actually route messages through their converters while preserving scalar behavior. Add automated integration-level unit tests with mocked model/service calls for the affected direct evaluators and at least representative legacy and non-legacy RAI paths; this would also catch wiring errors such as an initialized validator never being called.

class TestHoistMessagesToConversation:

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• This swap weakens validation for the new bare-message path: MessagesOrQueryResponseInputValidator checks only each item's type and role, whereas the previous ConversationValidator also requires non-empty, correctly typed content. A user/assistant pair with missing content now passes validation and is converted to empty query/response strings instead of raising an input error. Preserve the conversation-level structural validation when accepting messages=.

        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py:123

• This unconditional base-class hoist also enables messages= for UngroundedAttributesEvaluator, which inherits this method but explicitly supports single-turn query/response/context only (_ungrounded_attributes.py:13-16,64-82). The base converter will silently project every message pair into calls for that evaluator, despite the PR explicitly leaving evaluators without conversation overloads out of scope. Please opt the intended RAI evaluators in rather than applying the normalization to every subclass.

        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:20

• ConversationValidator is no longer referenced after the validator swap, so this import triggers the package's unused-import lint check. Remove it from the import list.

This issue also appears on line 99 of the same file.

    ConversationValidator,

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• The new validator's messages branch validates roles but not the required, non-empty content field that ConversationValidator previously enforced. Consequently malformed assistant messages pass _real_call validation and reach the fluency prompt with an empty or invalid response. Please retain the existing conversation message validation for this new input shape.

        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• The newly initialized _validator is never invoked anywhere in SimilarityEvaluator, so this bare-message path bypasses the validation the assignment appears intended to add. Validate the original kwargs before hoisting so invalid messages values produce the expected EvaluationException rather than failing later in the conversation converter.

        hoist_messages_to_conversation(kwargs)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• The newly initialized validator is never invoked by SimilarityEvaluator (the inherited _real_call does not use _validator). Consequently malformed bare input bypasses the intended checks; for example, messages=[] is hoisted, converted to zero turns, and returns {} instead of raising EvaluationException. Validate the messages/conversation path before hoisting while retaining the existing scalar-specific error checks.

        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py:123

• Applying this normalization in the shared RAI base changes more evaluators than the eight listed in the PR: GroundednessProEvaluator and UngroundedAttributesEvaluator also inherit this method. In particular, UngroundedAttributesEvaluator is documented as single-turn and requires context, but messages=[...] without context is now converted by _derive_conversation_converter into a call with the synthetic string "{}", bypassing that required input. Restrict the hoist to the intended evaluators, or explicitly define and validate the semantics for every affected subclass.

        # Normalize a bare ``messages=[...]`` kwarg (plus optional scalar ``context``
        # / ``ground_truth`` / ``tool_definitions``) into ``conversation={...}`` so
        # RAI safety evaluators route messages-shape input through the
        # ``_evaluate_conversation`` path instead of failing kwarg matching.
        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/tests/unittests/test_conversation_input.py:24

• These tests exercise only the helper in isolation; none invokes an affected evaluator with messages=. They therefore cannot detect missing validator wiring, interaction with singleton extraction, or the extra subclasses affected through RaiServiceEvaluatorBase. Add representative/parameterized evaluator-level tests for the five Prompty evaluators and the RAI path, including adjuncts and invalid empty messages.

class TestHoistMessagesToConversation:

sdk/evaluation/azure-ai-evaluation/dev_requirements.txt:24

• This prevents CI from exercising OpenAI 3.x even though setup.py still permits openai>=1.108.0 with no upper bound. A production incompatibility with the new major could therefore ship undetected. Please migrate the test proxy to httpx2 so the supported major remains covered, or temporarily cap the runtime dependency as well until compatibility is verified.

openai<3.0

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:17

• ConversationValidator is no longer referenced after the validator swap, so this import triggers the package's unused-import lint check. Remove it from the import list.

    ConversationValidator,

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:20

• ConversationValidator is no longer referenced after the validator swap. Leaving this import causes the package's pylint validation to report an unused import; remove it from the import list.

    ConversationValidator,

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• The newly initialized validator is never invoked in this class. Consequently, SimilarityEvaluator(messages=[]) is hoisted to an empty conversation and the inherited _real_call returns {} instead of raising the validator's EvaluationException; malformed message entries likewise bypass the new structural checks. Validate kwargs before hoisting (as Relevance and Fluency do in _real_call).

        hoist_messages_to_conversation(kwargs)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• The new validator's messages= branch does not perform the content validation that the previous ConversationValidator applies: entries with missing or empty content pass and are converted to empty responses. That makes bare messages behave differently from the equivalent conversation={"messages": ...} input despite this being a plumbing-only change. Please make the shared messages validator reuse the full conversation message validation before adopting it here.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• The replacement validator makes the new messages= path less strict than the existing conversation= path. MessagesOrQueryResponseInputValidator only checks that each entry is a dict with a valid role, so messages with missing/empty/invalid content pass here; ConversationValidator rejects those shapes, and the converter can otherwise score "". Since this is intended as routing-only normalization, please preserve the full conversation message validation (for example, by delegating the messages branch to the conversation validator after wrapping it).

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (4)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• This validator swap weakens the existing conversation validation for the new path. MessagesOrQueryResponseInputValidator validates item type and role but does not require non-empty content or validate structured content as ConversationValidator does. A bare user/assistant pair with missing content therefore passes and is scored as an empty response, whereas the equivalent conversation= payload is rejected. Normalize before validation and reuse ConversationValidator, or make the shared messages validator delegate to the same conversation validation.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:96

• This validator is never invoked: SimilarityEvaluator has no _real_call validation hook, and the base conversion path does not consult _validator. As a result, the new initialization does not validate bare messages at all despite the comment, so malformed payloads bypass the intended checks. Wire validation into the execution path (while preserving conversation-equivalent validation) or remove the dead field/imports.

        # Initialize input validator — accepts messages OR query/response(/ground_truth).
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.SIMILARITY_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_conversation_input.py:22

• The new input shape is accepted only dynamically. The affected evaluators' public __call__ overloads still expose only scalar inputs and conversation=, so mypy/Pyright reports evaluator(messages=[...]) as having no matching overload and generated API documentation omits the newly supported shape. Add typed messages overloads, including the supported adjuncts, to the direct evaluators and inherited RAI evaluator APIs.

def hoist_messages_to_conversation(kwargs: Dict[str, Any]) -> Dict[str, Any]:
    """Promote a bare ``messages=[...]`` kwarg into a
    ``conversation={"messages": [...], ...}`` dict so the base
    ``_derive_conversation_converter`` extracts per-turn ``query``/``response``
    for the judge.

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• This validator swap weakens the existing conversation validation for the new path. MessagesOrQueryResponseInputValidator checks only that each item is a dict with a valid role and that user/assistant roles exist; unlike ConversationValidator._validate_input_messages_list, it never requires non-empty content or validates structured content. Consequently, bare messages such as [{'role': 'user'}, {'role': 'assistant'}] now pass validation and are converted to empty query/response strings for scoring, while the equivalent conversation= input is rejected. Normalize before validation and reuse ConversationValidator, or make the shared messages validator delegate to the same conversation validation.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• The new validator is never invoked in this class. Unlike Relevance and Fluency, Similarity has no _real_call override, so messages=[] is hoisted, the base converter returns an empty list, and EvaluatorBase returns {} instead of rejecting the invalid input; malformed message entries can likewise escape the intended validation. Validate the bare messages shape before hoisting while preserving the existing scalar required-field checks.

        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:174

• Runtime support for messages= is not reflected in the public __call__ overloads or keyword documentation. Since type checkers use the overload signatures, a valid call such as evaluator(messages=[...]) is still reported as an unexpected argument, and generated API documentation will not expose this feature. Add a messages overload (including the supported adjuncts) to each affected direct and RAI evaluator.

    def _convert_kwargs_to_eval_input(self, **kwargs):
        """Normalize a bare ``messages=[...]`` kwarg (plus optional scalar
        ``context`` / ``ground_truth`` / ``tool_definitions``) into
        ``conversation={...}`` so the base ``_derive_conversation_converter``
        can extract per-turn q/r for the judge. Shared with other evaluators
        via ``hoist_messages_to_conversation``."""
        hoist_messages_to_conversation(kwargs)

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:96

• The new validator is never invoked: this class has no _real_call validation override, and neither EvaluatorBase nor PromptyEvaluatorBase calls self._validator. Thus malformed bare messages (including turns with missing content) still reach conversion/the model, so this initialization does not provide the stated validation. Either wire validation into the call path as Relevance/Fluency do or remove the dead validator and validate during conversion.

        # Initialize input validator — accepts messages OR query/response(/ground_truth).
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.SIMILARITY_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• This validator's messages branch only checks that entries are dicts with valid roles; unlike ConversationValidator._validate_conversation, it never requires non-empty content. Consequently, RelevanceEvaluator(messages=[{"role": "user"}, {"role": "assistant"}]) now passes validation and the converter evaluates empty strings, while the equivalent conversation= input is rejected. Please make bare messages go through the same message-content validation as the conversation path.

        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• This changes bare-message validation semantics: MessagesOrQueryResponseInputValidator validates roles but not the required, non-empty content field that ConversationValidator checks. Fluency therefore sends empty content to the judge for malformed messages= input even though the equivalent conversation= input raises an EvaluationException. Please route bare messages through the existing full conversation-message validation.

        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (7)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:160

• Bare messages takes this early conversation branch before the existing required-field checks. If neither the scalar adjunct nor assistant turns provide ground_truth, _derive_conversation_converter supplies ground_truth="", so Similarity evaluates without the reference that the scalar path rejects as missing; the newly assigned validator is not invoked anywhere in this class either. Validate the messages shape and require per-turn/top-level ground truth before returning through this branch.

        hoist_messages_to_conversation(kwargs)
        if kwargs.get("conversation") is not None:
            return super()._convert_kwargs_to_eval_input(**kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• This validator is weaker than the replaced ConversationValidator: its messages branch checks only dictionary type and roles, not that content exists, is valid, or is non-empty. Consequently bare messages such as [{"role":"user"}, {"role":"assistant"}] now pass validation and are scored as empty strings, whereas the equivalent conversation input is rejected. Preserve the conversation-level structural validation for the new bare-messages path.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• The replacement validator does not validate message content, unlike ConversationValidator; it accepts missing or empty content as long as user and assistant roles exist. The converter then turns those messages into an empty response and Fluency sends it to the judge. The bare-messages path should retain the same structural/content validation as the existing conversation path.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_retrieval/_retrieval.py:158

• This also accepts bare messages with no top-level or per-turn context, even though Retrieval's scalar overload requires context. In that case the conversation converter produces the literal string "{}" as context and the model returns a score without any retrieved material to assess. Reject this shape unless global or query-turn context is available.

        hoist_messages_to_conversation(kwargs)
        return super()._convert_kwargs_to_eval_input(**kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_response_completeness/_response_completeness.py:151

• Bare messages without scalar or per-assistant ground_truth are now routed through the conversation converter, which inserts ground_truth="". _do_eval only checks that the key exists, so this bypasses the established missing-ground-truth error and asks the model to assess completeness without a reference. Require ground truth before accepting this new input shape.

        hoist_messages_to_conversation(kwargs)
        return super()._convert_kwargs_to_eval_input(**kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_conversation_input.py:22

• The runtime normalization enables a new public call shape, but none of the affected evaluators' __call__ overloads or docstrings declare messages (or its adjuncts). Static type checkers therefore report evaluator(messages=...) as having no matching overload, and generated API documentation does not expose the feature. Add the messages overload/documentation to each affected public evaluator while accounting for overload introspection in EvaluatorBase.

def hoist_messages_to_conversation(kwargs: Dict[str, Any]) -> Dict[str, Any]:
    """Promote a bare ``messages=[...]`` kwarg into a
    ``conversation={"messages": [...], ...}`` dict so the base
    ``_derive_conversation_converter`` extracts per-turn ``query``/``response``
    for the judge.

sdk/evaluation/azure-ai-evaluation/dev_requirements.txt:24

• setup.py allows openai>=1.108.0 with no upper bound, but this global development pin forces every package test environment onto 2.x. Thus the stated OpenAI 3 runtime compatibility is never exercised in CI, and v3-only production regressions can ship undetected. Update the test proxy for v3 or retain a dedicated v3 test job rather than excluding a supported major version from all tests.

openai<3.0

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (5)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_rai_svc_eval.py:127

• The legacy short-circuit bypasses the base class's mutual-exclusion check. After hoisting, a call containing messages plus query/response returns all fields here; _do_eval sees response and silently evaluates the scalar pair while ignoring the messages. An explicit conversation plus bare messages is likewise silently accepted. Reject these mixed shapes before returning [kwargs], matching the non-legacy path and the evaluator contract.

        if self._use_legacy_endpoint and "conversation" in kwargs and kwargs["conversation"] is not None:
            # Legacy endpoint: pass conversation through intact so _evaluate_conversation
            # can send all messages in a single API call (pre-sync-migration behavior).
            return [kwargs]

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_relevance/_relevance.py:99

• Replacing ConversationValidator here weakens validation for the new bare-messages path. MessagesOrQueryResponseInputValidator checks only that each item is a dict with a valid role; it does not enforce the required/non-empty content field that ConversationValidator._validate_input_messages_list enforces. Because _real_call validates before this class hoists, inputs such as messages=[{"role": "user"}, {"role": "assistant", "content": "a"}] now pass validation and are scored with an empty query, while the equivalent conversation={"messages": ...} is rejected. Please make the shared messages validator apply the full conversation message validation, or hoist before retaining the existing validator.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(error_target=ErrorTarget.RELEVANCE_EVALUATOR)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_fluency/_fluency.py:93

• This validator swap makes bare messages less strictly validated than the equivalent conversation. The new validator accepts missing or empty content fields, whereas ConversationValidator rejects them; the base converter then defaults missing content to "" and Fluency can score invalid input. Please reuse the full conversation-message validation for the messages branch so this plumbing alias preserves existing validation semantics.

        # Initialize input validator — accepts messages OR query/response.
        self._validator = MessagesOrQueryResponseInputValidator(
            error_target=ErrorTarget.FLUENCY_EVALUATOR,
            requires_query=False,
        )

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_similarity/_similarity.py:158

• The newly initialized validator is never called by SimilarityEvaluator: unlike Relevance and Fluency, this class has no _real_call override, and the base class does not invoke _validator. Consequently SimilarityEvaluator(messages=[]) bypasses the intended empty-input guard and returns {}. Invoke the validator for a bare messages input before hoisting; restricting it to this branch preserves the existing scalar error messages.

        hoist_messages_to_conversation(kwargs)

sdk/evaluation/azure-ai-evaluation/dev_requirements.txt:24

• This makes CI install OpenAI 2.x while setup.py still advertises the unbounded runtime requirement openai>=1.108.0. Customers will therefore resolve 3.x, but none of this package's tests will exercise that supported major, so runtime incompatibilities can ship undetected. Please update the proxy patcher for httpx2 (or add a separate 3.x test environment) rather than globally pinning tests below a version that remains supported for end users; otherwise the runtime dependency must be capped consistently.

openai<3.0