feat(agents): add initialization_parameters to evaluator YAML config

[Ilya Matiach (imatiach-msft)]

Closes #9624

Problem

Evaluators that require configuration parameters (e.g. builtin.regex_match which requires patterns) couldn't be fully specified from the optimize YAML config. EvaluatorRef only had name and version.

Change

• Added initialization_parameters map[string]any to EvaluatorRef in the YAML config struct
• Wired it through inline per-evaluator into the OptimizeRequest evaluators array (the C# API reads initialization_parameters per-entry, not from a top-level map)
• Fixed MarshalYAML: an evaluator with only initialization_parameters (no version/local_uri) was previously emitted as a plain scalar, silently dropping the params on round-trip
• Added TestConfig_RoundTrip_EvaluatorInitializationParameters covering plain, init-params-only, and version+init-params cases

Usage

evaluators:
  - name: builtin.regex_match
    initialization_parameters:
      patterns:
        - "(?i)Answer:\\s*{{ground_truth}}"
  - builtin.task_adherence

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
19 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Note

This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (3)

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:496

• This introduces a separate user-facing YAML/API option that is absent from the PR description and has no parsing, forwarding, omission, or bounds-validation tests. Either remove the unrelated max_concurrent_agent_runs change from this PR, or document its contract and add validation/tests before exposing it. [azd-code-reviewer]

}

// Read reads a YAML config file (eval or optimize format).
func Read(path string) (*Config, error) {

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:102

• EvaluatorList.MarshalYAML still emits an evaluator as a scalar whenever version and local URI are empty, so an entry containing only initialization_parameters is rewritten as just its name and silently loses the new data through Config.Write. Include this field when deciding whether an entry requires mapping form, and add a write/read round-trip case without a version. [azd-code-reviewer]

	// InitializationParameters holds evaluator-specific configuration parameters
	// such as regex_match's ``patterns``. Sent inline per-evaluator in the
	// evaluators array of the API request.
	InitializationParameters map[string]any `yaml:"initialization_parameters,omitempty" json:"-"`

cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_config.go:259

• The first line of mergeEvaluators' comment was deleted, leaving this paragraph starting with the fragment “exists in base” and no longer identifying what is being described. Restore the deleted // mergeEvaluators appends add to base, skipping entries whose name already line. [azd-code-reviewer]

// new entries from add. Used to layer --evaluator flags on top of config

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:100

• [azd-code-reviewer] EvaluatorList.MarshalYAML still emits the scalar form whenever version and local URI are empty. An evaluator containing only a name plus this new map is therefore written as just its name by Write/WriteEvalConfig, silently dropping all initialization parameters. Treat a ref as scalar only when InitializationParameters is also empty, and cover this with a write/read round-trip test.

	InitializationParameters map[string]any `yaml:"initialization_parameters,omitempty" json:"-"`

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (4)

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/models.go:84

• [azd-code-reviewer] StartOptimize marshals this field inside each evaluators[] object, but the optimizer contract described by this PR reads EvaluatorInitParamsMap from the top-level request. These values therefore will not reach the optimizer. Keep the wire EvaluatorRef as name/version, add and populate the top-level request map, and assert the serialized request shape.

	InitializationParameters map[string]any `json:"initialization_parameters,omitempty"`

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:100

• [azd-code-reviewer] EvaluatorList.MarshalYAML still emits an evaluator as a scalar whenever version and local_uri are empty. An evaluator that only sets this new map is therefore written as just its name by Write, silently dropping its initialization parameters on a write/read round trip. Include non-empty initialization parameters in the structured-entry condition and cover that round trip.

	InitializationParameters map[string]any `yaml:"initialization_parameters,omitempty" json:"-"`

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:94

• [azd-code-reviewer] The previous two-line EvaluatorRef comment remains directly above this addition, so the Go doc now repeats itself and first lists an outdated set of fields. Replace the four lines with the updated two-line comment once.

This issue also appears on line 100 of the same file.

// EvaluatorRef describes an evaluator. It can be a simple string name or a
// structured entry with name, version, local_uri, and initialization_parameters.

cli/azd/extensions/azure.ai.agents/internal/cmd/optimize_config.go:258

• [azd-code-reviewer] The opening line of the mergeEvaluators comment was deleted, leaving this documentation starting mid-sentence with “exists in base.” Restore the removed mergeEvaluators appends... line so the comment is complete and names the function it documents.

// exists in base (case-sensitive). Order is preserved: base first, then any

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/models.go:84

• The optimize service expects these values in the top-level OptimizeRequest.EvaluatorInitParamsMap, not inside each evaluators[] entry. StartOptimize will currently serialize initialization_parameters on the evaluator and never emit the top-level map, so the optimizer cannot resolve the required parameters. Keep the wire EvaluatorRef name/version-only and populate the request-level map in ToRequest. (azd-code-reviewer)

	InitializationParameters map[string]any `json:"initialization_parameters,omitempty"`

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/optimize_api/models.go:84

• [azd-code-reviewer] This serializes initialization_parameters inside each evaluators[] item, but the optimizer contract described by this PR expects a top-level evaluatorInitParamsMap. OptimizeRequest has no such field, so the backend will not receive the map and evaluators such as builtin.regex_match still cannot be initialized. Keep this value YAML-only on the evaluator reference, add/populate the top-level request map keyed by evaluator, and assert the marshaled request shape.

	InitializationParameters map[string]any `json:"initialization_parameters,omitempty"`

cli/azd/extensions/azure.ai.agents/internal/pkg/agents/opt_eval/yaml.go:94

• [azd-code-reviewer] There are now two consecutive EvaluatorRef doc comments, and the first still claims structured entries only support name, version, and local_uri. Remove the obsolete first comment so the public type has one accurate description.

// EvaluatorRef describes an evaluator. It can be a simple string name or a
// structured entry with name, version, local_uri, and initialization_parameters.

[John Miller (therealjohn)]

approved scenario

[Travis Angevine (trangevi)]

/check-enforcer override