fix(workflow): report actual agent kind in analyze text output - #383
Merged
Paulo Lacerda (placerda) merged 1 commit intoAug 7, 2026
Merged
Conversation
`agentops workflow analyze --format text` hardcoded the "Agent target" row
as "Foundry prompt agent (`name:version`)." regardless of the project's
actual target. `_friendly_foundry_eval_text` discarded its `text` argument
for that check and returned a constant, so hosted-agent / azd-recipe
projects were mislabelled. The markdown renderer emits the computed
explanation unmodified, so the two formats contradicted each other.
Remove the hardcoded branch so the text renderer passes the computed
explanation through, matching markdown.
Also extend `_soften_text` to cover every raw kind emitted by
`classify_agent` instead of only `foundry_prompt`. This fixes the related
leak called out in the issue: the agentops.yaml signal detail
(`f"agentops.yaml targets {target.kind}."`) rendered the raw token
`foundry_hosted` in text output. Labels are reused from the existing
`eval_analysis._friendly_target` map rather than invented.
Before (text, hosted agent):
ok Agent target Foundry prompt agent (`name:version`).
ok Target agentops.yaml targets foundry_hosted.
After (text, hosted agent):
ok Agent target Found azd eval recipe at `eval.yaml`.
ok Target agentops.yaml targets Foundry hosted agent.
Markdown output is unchanged and was already correct.
Adds three regression tests covering the hosted-agent case, the
prompt-agent case (no regression), and the softened signal detail.
Fixes #370
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This was referenced Aug 7, 2026
Paulo Lacerda (placerda)
added a commit
that referenced
this pull request
Aug 7, 2026
`eval analyze` labelled every dataset containing an `expected` column as `model quality`, because `_scenario_hint` inspected only the dataset column names and never received the resolved target kind. The reported symptom is real, and it is broader than issue #363 describes: prompt agents were mislabelled too, not just hosted agents. target scenario_hint (before) scenario_hint (after) foundry_hosted model_quality (wrong) conversational foundry_prompt model_quality (wrong) conversational model_direct model_quality (right) model_quality The docs are correct and stay as they are. Three independent sources already agree that an agent target answering `input` is conversational rather than model quality, so this aligns the code with them rather than the reverse: - docs/concepts.md scopes "Model quality" to `model:<deployment>` + expected. - AGENTS.md distinguishes `model:<deployment>` from "any prompt/hosted agent". - core/evaluators.py already branches on `target.kind == "model_direct"` when selecting evaluators, so the engine never agreed with the label. `scenario_hint` is reporting-only and does not change which evaluators run, which is why the mislabel went unnoticed. An unknown target keeps the previous `model_quality` label, since a missing or unparseable config yields both `target_kind=None` and an empty column set and cannot reach that branch. While here, fix the same `_soften_text` leak that #370 fixed in the sibling renderer: `eval analyze --format text` printed the raw kind string, as in "Project: foundry_hosted evaluation setup". Both renderers now derive their labels from one shared table so they cannot drift apart again. Also remove the two unreachable fallbacks in `_foundry_eval_rows`, deliberately left alone in #383 to keep that PR focused. Every path that selects a Foundry eval runner populates exactly two reasons: the azd branch builds a two-item list inline, and the cloud branch is gated on `official_support.eligible`, which always carries a hardcoded two-reason tuple. `OFFICIAL_EVAL_RUNNER` is never assigned in this function. The rows are now zipped with their labels, so they always describe the real analysis and cannot raise IndexError if the invariant ever changes. The removed `else "Foundry prompt agent."` fallback was also an instance of the #370 bug, hardcoding a prompt-agent label on a path a hosted agent can reach. Tests: 5 added, all verified failing against the unfixed code first. Full suite 1116 passed, 5 skipped. Fixes #363 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: bcb9c0b6-d506-46dc-90d2-8120413166ee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agentops workflow analyze --format texthardcoded the Agent target row asFoundry prompt agent (\name:version`).regardless of the project's actual target._friendly_foundry_eval_textdiscarded itstext` argument for that check and returned a constant, so hosted-agent / azd-recipe projects were mislabelled as prompt agents.The markdown renderer emits the computed
explanationunmodified and was already correct, so the two output formats contradicted each other.Fixes #370
Changes
1.
_friendly_foundry_eval_text— remove the hardcoded branch (the fix suggested in the issue)The text renderer now passes the computed explanation through, matching markdown.
2.
_soften_text— cover every target kind, not justfoundry_promptThis addresses the Related note in the issue.
_agentops_config_factsemitsf"agentops.yaml targets {target.kind}."into a signal detail, which flows through_soften_text. Because onlyfoundry_promptwas mapped, hosted projects leaked the raw tokenfoundry_hostedinto user-facing output.Labels are reused verbatim from the existing
eval_analysis._friendly_targetmap rather than invented, so vocabulary stays consistent across the two analyze commands.Before / after
Hosted-agent project (
agentops.yamlwith an agent URL +execution: azd, plus aneval.yamlazd recipe):Before (
--format text):After (
--format text):Markdown (unchanged in both — this is the correct value the text renderer was throwing away):
Prompt-agent projects are unaffected: both formats still report
Agent target is a Foundry prompt agent (\name:version`).`Tests
Three regression tests added to
tests/unit/test_workflow_analysis.py:test_agent_target_text_matches_markdown_for_hosted_agenttest_agent_target_text_matches_markdown_for_prompt_agenttest_hosted_agent_kind_is_softened_in_text_signals_soften_textleak — asserts rawfoundry_hostednever reaches outputVerified failing-first: with the source reverted to
HEADall three fail (3 failed / 14 passed); with the fix applied all pass (17 passed).Validation
pytest tests/unit/test_workflow_analysis.pypytest tests/(full suite)ruff check src/ tests/(v0.15.5, matchinguv.lock)mypy src/agentops/services/workflow_analysis.py --ignore-missing-importsNotes for reviewers
_soften_text) goes slightly beyond the issue's literal one-line suggestion. It is justified by the issue's own Related paragraph, fixes a second real user-visible leak, and is covered by its own test. Happy to split it out if preferred."Foundry prompt agent."fallback in_foundry_eval_rows(theelsewhenofficial_eval_reasonsis empty) is unreachable —selectedrequires an azd/cloud/official runner, and both branches that assign those runners always populate exactly two reasons. Touching it would add untestable code. Flagging it as latent-but-dead rather than silently editing it.eval analyzewas checked for the same bug:eval_analysis.pyalready renders the target correctly via_friendly_target, so no change was needed there.