Use iloc when computing faithfulness metric

[harupy]

🛠 DevTools 🛠

Install mlflow from this PR

pip install git+https://github.com/mlflow/mlflow.git@refs/pull/11117/merge

Checkout with GitHub CLI

gh pr checkout 11117

Related Issues/PRs

Resolve #11108

What changes are proposed in this pull request?

How is this PR tested?

• Existing unit/integration tests
• New unit/integration tests
• Manual tests

Does this PR require documentation update?

• No. You can skip the rest of this section.
• Yes. I've updated:
  • Examples
  • API references
  • Instructions

Release Notes

Is this a user-facing change?

• No. You can skip the rest of this section.
• Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

• area/artifacts: Artifact stores and artifact logging
• area/build: Build and test infrastructure for MLflow
• area/deployments: MLflow Deployments client APIs, server, and third-party Deployments integrations
• area/docs: MLflow documentation pages
• area/examples: Example code
• area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
• area/models: MLmodel format, model serialization/deserialization, flavors
• area/recipes: Recipes, Recipe APIs, Recipe configs, Recipe Templates
• area/projects: MLproject format, project running backends
• area/scoring: MLflow Model server, model deployment tools, Spark UDFs
• area/server-infra: MLflow Tracking server backend
• area/tracking: Tracking Service, tracking client APIs, autologging

Interface

• area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
• area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
• area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
• area/windows: Windows support

Language

• language/r: R APIs and clients
• language/java: Java APIs and clients
• language/new: Proposals for new client languages

Integrations

• integrations/azure: Azure and Azure ML integrations
• integrations/sagemaker: SageMaker integrations
• integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

• rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
• rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
• rn/feature - A new user-facing feature worth mentioning in the release notes
• rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
• rn/documentation - A user-facing documentation change worth mentioning in the release notes

[github-actions]

Documentation preview for 83ccbdc will be available when this CircleCI job completes successfully.

• Top page
• Changed pages

More info

• Ignore this comment if this PR does not change the documentation.
• It takes a few minutes for the preview to be available.
• The preview is updated when a new commit is pushed to this PR.
• This comment was created by https://github.com/mlflow/mlflow/actions/runs/7897503095.

[harupy]

It looks like the existing code assumes that inputs, outputs, and eval_values[arg] have the same indices and throws when they don't.

mlflow/mlflow/metrics/genai/genai_metric.py

Lines 277 to 300 in cc184ee

	grading_payloads = []
	for indx, (input, output) in enumerate(zip(inputs, outputs)):
	try:
	arg_string = _format_args_string(grading_context_columns, eval_values, indx)
	except Exception as e:
	raise MlflowException(
	f"Values for grading_context_columns are malformed and cannot be "
	f"formatted into a prompt for metric '{name}'.\n"
	f"Required columns: {grading_context_columns}\n"
	f"Values: {eval_values}\n"
	f"Error: {e!r}\n"
	f"Please check the following: \n"
	"- predictions and targets (if required) are provided correctly\n"
	"- grading_context_columns are mapped correctly using the evaluator_config "
	"parameter\n"
	"- input and output data are formatted correctly."
	)
	grading_payloads.append(
	evaluation_context["eval_prompt"].format(
	input=(input if include_input else None),
	output=output,
	grading_context_columns=arg_string,
	)
	)

In the new code, they don't have to have the same indices.

[B-Step62]

nit: the error is rather because of index not starting from 0 I guess?

for indx, (input, output) in enumerate(zip(inputs, outputs)):
            try:
                arg_string = _format_args_string(grading_context_columns, eval_values, indx)

The indx always starts from 0 (cuz we do enumerate) and eval_values[0] will raise KeyError if it doesn't have index 0.

[harupy]

Exactly, reset_index(drop=True) should also work as @ai-learner-00 explained in #11108

[B-Step62]

yup I think the current solution is fine, plz feel free to merge. Just wanted to make sure the ^comment is accurate:)

[B-Step62]

LGTM!