Docs/code drift: `eval analyze` reports `model quality` for hosted-agent targets that the docs classify as `Conversational`

[placerda]

Summary

The scenario label reported by agentops eval analyze disagrees with the scenario table published in the Evaluation docs. A Foundry hosted agent target with a dataset containing only input / expected is reported as model quality, but the docs define Model quality as requiring a model:<deployment> target.

Version: agentops-accelerator 0.8.1

What the docs say

From Datasets and scenarios on https://azure.github.io/agentops/evaluation/:

Scenario	Signal in the row
Model quality	model:<deployment> target plus expected
Conversational	input plus expected

By that table, a hosted-agent target with input + expected rows should be classified as Conversational.

What the code does

agentops/services/eval_analysis.py, _scenario_hint() (around line 410) only inspects dataset column names. It never receives or consults the resolved target kind:

def _scenario_hint(dataset_columns: Set[str], text: str) -> str:
    if {"tool_calls", "tool_definitions"} & dataset_columns:
        return "agent_workflow"
    if "context" in dataset_columns:
        return "rag"
    if "conversation" in dataset_columns or "turns" in dataset_columns:
        return "conversational"
    if "expected" in dataset_columns:
        return "model_quality"
    ...

Because the expected check precedes any target-kind consideration, every dataset with an expected column is labeled model_quality regardless of whether the target is a model deployment, a prompt agent, or a hosted agent. The conversational branch is only reachable via conversation / turns columns, which is a different signal than the one the docs advertise.

Reproduction

agentops.yaml:

version: 1
agent: https://<acct>.services.ai.azure.com/api/projects/<proj>/agents/<agent>/versions/11
dataset: .agentops/data/helpdesk-smoke.jsonl
protocol: responses

Dataset rows (columns: id, input, expected):

{"id": "1", "input": "Diagnose why the user cannot sign in.", "expected": "Explain that the token expired and tell the user to sign out and back in."}

Run:

agentops eval analyze

Observed output:

Project: foundry_hosted evaluation setup (model quality)

Readiness
  target    Foundry hosted agent
  scenario  model quality
  ...
  ok Scenario  Likely scenario: model quality.

The same report correctly resolves target as Foundry hosted agent, so the target kind is known at that point. It just isn't used by the scenario heuristic.

Impact

Low, but non-zero.

Functionally this appears harmless: agentops/core/evaluators.py states in its module docstring that "There is no user-facing scenario concept," and evaluator auto-selection is driven by target kind plus dataset shape rather than by scenario_hint. The label does not change which evaluators run.

The cost is trust in the reporting surface. eval analyze is presented as the step you run to understand your setup before your first run, so a label that contradicts the published table sends readers looking for a misconfiguration that does not exist.

Suggested fix

Either of these would close the gap:

1. Align the code with the docs. Pass the resolved target kind into _scenario_hint() and gate the model_quality branch on a model-direct target, falling through to conversational for prompt/hosted agent targets with input + expected.
2. Align the docs with the code. Update the Datasets and scenarios table to describe the actual precedence (column-only, expected wins over the conversational case), and drop the model:<deployment> qualifier from the Model quality row.

Option 1 seems closer to the documented intent, since the analyze report already surfaces the target kind on the adjacent line.

Related, smaller gap

While tracing this, one behavior that is only discoverable in code is the per-target-kind evaluator split described in the core/evaluators.py docstring:

• Model-direct targets add Similarity + F1Score
• Agent targets add Similarity + ResponseCompleteness

The docs describe both as "answer-quality judges" without distinguishing them. Documenting the concrete default sets per target kind would make it easier to predict a run before executing it. Happy to split that into its own issue if you prefer to keep this one focused.

[placerda]

Fixed in #387 (merged to develop as 4cb561a).

Root cause was broader than reported: _scenario_hint inspected only dataset column names, so any dataset with an expected column was labelled model quality -- Foundry prompt agents were mislabelled too, not just hosted agents. The docs were correct; core/evaluators.py already branched on target.kind == 'model_direct' for the real evaluator selection, so _scenario_hint was the outlier.

Also fixed in the same PR: --format text leaked the raw kind string (Project: foundry_hosted evaluation setup), the same _soften_text gap that #370 closed in the sibling renderer.