Skip to content

fix(workflow): report actual agent kind in analyze text output - #383

Merged
Paulo Lacerda (placerda) merged 1 commit into
developfrom
fix/workflow-analyze-agent-target
Aug 7, 2026
Merged

fix(workflow): report actual agent kind in analyze text output#383
Paulo Lacerda (placerda) merged 1 commit into
developfrom
fix/workflow-analyze-agent-target

Conversation

@placerda

Copy link
Copy Markdown
Contributor

Summary

agentops workflow analyze --format text hardcoded the Agent target row as Foundry prompt agent (\name:version`).regardless of the project's actual target._friendly_foundry_eval_textdiscarded itstext` argument for that check and returned a constant, so hosted-agent / azd-recipe projects were mislabelled as prompt agents.

The markdown renderer emits the computed explanation unmodified and was already correct, so the two output formats contradicted each other.

Fixes #370

Changes

1. _friendly_foundry_eval_text — remove the hardcoded branch (the fix suggested in the issue)

def _friendly_foundry_eval_text(check: str, text: str) -> str:
    if check == "Evaluators":
        return _friendly_evaluator_list(text.split(", "))
    return _soften_text(text)

The text renderer now passes the computed explanation through, matching markdown.

2. _soften_text — cover every target kind, not just foundry_prompt

This addresses the Related note in the issue. _agentops_config_facts emits f"agentops.yaml targets {target.kind}." into a signal detail, which flows through _soften_text. Because only foundry_prompt was mapped, hosted projects leaked the raw token foundry_hosted into user-facing output.

#: Raw target kinds emitted by :func:`classify_agent` mapped to display names.
#: Ordered longest-key-first so no key is a prefix of another when substituted.
_TARGET_KIND_LABELS = (
    ("foundry_prompt", "Foundry prompt agent"),
    ("foundry_hosted", "Foundry hosted agent"),
    ("model_deployment", "model deployment"),
    ("model_direct", "direct model"),
    ("http_json", "HTTP/JSON agent"),
)

Labels are reused verbatim from the existing eval_analysis._friendly_target map rather than invented, so vocabulary stays consistent across the two analyze commands.

Before / after

Hosted-agent project (agentops.yaml with an agent URL + execution: azd, plus an eval.yaml azd recipe):

Before (--format text):

ok   Agent target  Foundry prompt agent (`name:version`).
ok   Target        agentops.yaml targets foundry_hosted.

After (--format text):

ok   Agent target  Found azd eval recipe at `eval.yaml`.
ok   Target        agentops.yaml targets Foundry hosted agent.

Markdown (unchanged in both — this is the correct value the text renderer was throwing away):

| [x] | Agent target | Found azd eval recipe at `eval.yaml`. |

Prompt-agent projects are unaffected: both formats still report Agent target is a Foundry prompt agent (\name:version`).`

Tests

Three regression tests added to tests/unit/test_workflow_analysis.py:

Test Covers
test_agent_target_text_matches_markdown_for_hosted_agent The #370 regression — asserts text == markdown and that the hardcoded string is gone
test_agent_target_text_matches_markdown_for_prompt_agent Guards against regressing the prompt-agent path
test_hosted_agent_kind_is_softened_in_text_signals The _soften_text leak — asserts raw foundry_hosted never reaches output

Verified failing-first: with the source reverted to HEAD all three fail (3 failed / 14 passed); with the fix applied all pass (17 passed).

Validation

Check Result
pytest tests/unit/test_workflow_analysis.py 17 passed
pytest tests/ (full suite) 1111 passed, 5 skipped
ruff check src/ tests/ (v0.15.5, matching uv.lock) All checks passed
mypy src/agentops/services/workflow_analysis.py --ignore-missing-imports Success, no issues

Notes for reviewers

  • Scope: change 2 (_soften_text) goes slightly beyond the issue's literal one-line suggestion. It is justified by the issue's own Related paragraph, fixes a second real user-visible leak, and is covered by its own test. Happy to split it out if preferred.
  • Intentionally not changed: the "Foundry prompt agent." fallback in _foundry_eval_rows (the else when official_eval_reasons is empty) is unreachable — selected requires an azd/cloud/official runner, and both branches that assign those runners always populate exactly two reasons. Touching it would add untestable code. Flagging it as latent-but-dead rather than silently editing it.
  • eval analyze was checked for the same bug: eval_analysis.py already renders the target correctly via _friendly_target, so no change was needed there.

`agentops workflow analyze --format text` hardcoded the "Agent target" row
as "Foundry prompt agent (`name:version`)." regardless of the project's
actual target. `_friendly_foundry_eval_text` discarded its `text` argument
for that check and returned a constant, so hosted-agent / azd-recipe
projects were mislabelled. The markdown renderer emits the computed
explanation unmodified, so the two formats contradicted each other.

Remove the hardcoded branch so the text renderer passes the computed
explanation through, matching markdown.

Also extend `_soften_text` to cover every raw kind emitted by
`classify_agent` instead of only `foundry_prompt`. This fixes the related
leak called out in the issue: the agentops.yaml signal detail
(`f"agentops.yaml targets {target.kind}."`) rendered the raw token
`foundry_hosted` in text output. Labels are reused from the existing
`eval_analysis._friendly_target` map rather than invented.

Before (text, hosted agent):
    ok   Agent target  Foundry prompt agent (`name:version`).
    ok   Target        agentops.yaml targets foundry_hosted.

After (text, hosted agent):
    ok   Agent target  Found azd eval recipe at `eval.yaml`.
    ok   Target        agentops.yaml targets Foundry hosted agent.

Markdown output is unchanged and was already correct.

Adds three regression tests covering the hosted-agent case, the
prompt-agent case (no regression), and the softened signal detail.

Fixes #370

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
@placerda
Paulo Lacerda (placerda) merged commit 893152e into develop Aug 7, 2026
12 checks passed
@placerda
Paulo Lacerda (placerda) deleted the fix/workflow-analyze-agent-target branch August 7, 2026 14:44
Paulo Lacerda (placerda) added a commit that referenced this pull request Aug 7, 2026
`eval analyze` labelled every dataset containing an `expected` column as
`model quality`, because `_scenario_hint` inspected only the dataset column
names and never received the resolved target kind.

The reported symptom is real, and it is broader than issue #363 describes:
prompt agents were mislabelled too, not just hosted agents.

    target          scenario_hint (before)   scenario_hint (after)
    foundry_hosted  model_quality  (wrong)   conversational
    foundry_prompt  model_quality  (wrong)   conversational
    model_direct    model_quality  (right)   model_quality

The docs are correct and stay as they are. Three independent sources already
agree that an agent target answering `input` is conversational rather than
model quality, so this aligns the code with them rather than the reverse:

- docs/concepts.md scopes "Model quality" to `model:<deployment>` + expected.
- AGENTS.md distinguishes `model:<deployment>` from "any prompt/hosted agent".
- core/evaluators.py already branches on `target.kind == "model_direct"` when
  selecting evaluators, so the engine never agreed with the label.

`scenario_hint` is reporting-only and does not change which evaluators run,
which is why the mislabel went unnoticed. An unknown target keeps the previous
`model_quality` label, since a missing or unparseable config yields both
`target_kind=None` and an empty column set and cannot reach that branch.

While here, fix the same `_soften_text` leak that #370 fixed in the sibling
renderer: `eval analyze --format text` printed the raw kind string, as in
"Project: foundry_hosted evaluation setup". Both renderers now derive their
labels from one shared table so they cannot drift apart again.

Also remove the two unreachable fallbacks in `_foundry_eval_rows`, deliberately
left alone in #383 to keep that PR focused. Every path that selects a Foundry
eval runner populates exactly two reasons: the azd branch builds a two-item
list inline, and the cloud branch is gated on `official_support.eligible`,
which always carries a hardcoded two-reason tuple. `OFFICIAL_EVAL_RUNNER` is
never assigned in this function. The rows are now zipped with their labels, so
they always describe the real analysis and cannot raise IndexError if the
invariant ever changes. The removed `else "Foundry prompt agent."` fallback was
also an instance of the #370 bug, hardcoding a prompt-agent label on a path a
hosted agent can reach.

Tests: 5 added, all verified failing against the unfixed code first. Full suite
1116 passed, 5 skipped.

Fixes #363

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant