[bot] Merge master/142bcf03 into rel/dev - #1744
Merged
Merged
Conversation
…g MAQL clauses
agentic_metric_skill's simulated-user reply (generate_simulated_response) is
what keeps a multi-turn metric-creation conversation going after the agent
asks a clarifying question -- it prompts an LLM to answer as the user, using
the fixture's expected_output.maql as its only source of truth.
The prompt told it to "reply briefly" with no instruction to preserve the
MAQL's structure. In practice it would silently drop a WHERE/filter clause,
or paraphrase a label id, whenever the agent's question didn't happen to ask
about that part directly -- so a well-behaved agent, faithfully following
the (already-wrong) simulated answer, still failed the eval.
Reproduced live twice against a real gdc-mic-ai-evaluation fixture
("Create a metric for total ecommerce spend", expects
SELECT {metric/spend_amount_-_cutcgco} WHERE {label/ecommerce_indicator_code}
= "1"):
1. Simulated reply dropped "_code" off ecommerce_indicator_code, anchoring
the agent on a sibling attribute that doesn't have that filter.
2. Simulated reply picked one of 3 metric options the agent offered and
said "please proceed with that" -- never mentioning the WHERE clause
that expected_output required, even though it had it in hand.
Confirmed via a 5x-repeated A/B test that this is a prompt problem, not a
model-capability one: swapping gpt-4o-mini for gpt-4o under the OLD prompt
did not fix it (still dropped the clause); the NEW prompt fixes it on the
ORIGINAL gpt-4o-mini (1/5 -> 5/5 runs preserving the exact filter).
Fix: instruct the simulating LLM to (a) ensure every clause of the expected
MAQL is eventually satisfied even if the agent's question didn't ask about
it, (b) quote field/label identifiers verbatim rather than paraphrase them,
and (c) proactively add a filter the agent's own offered options omitted.
Also drop "reply briefly" and raise max_tokens 150->300, since brevity was
part of what squeezed the filter clause out. This brings metric_skill's
simulated-user prompt in line with alert_skill's generate_simulated_alert_response,
which already passes structured facts + explicit "proactively tell the agent
X" instructions rather than one freely-paraphrased string -- not a new
pattern for this codebase.
Added a regression test asserting the sent prompt preserves clause-fidelity
language and the raised max_tokens. Full gooddata-eval suite: 272 passed
(9 pre-existing unrelated failures, confirmed identical on clean master
before this change -- missing openai extra in test env, and two unrelated
test files).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…er prompt Addresses CodeRabbit review comment on #1718: the regression test only checked for generic instruction words ("verbatim", "every clause"), not that expected_output["maql"] itself made it into the prompt -- a regression that stripped the metric/label reference or filter value entirely could still pass. Assert the exact MAQL string is present. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
GoodData has no admin-settable "default agent": when a conversation
doesn't name one, the platform picks whichever agent was last used or
last edited in that workspace. Orgs with multiple AI Hub agents (e.g.
one scoped to visualization only) can end up silently evaluating the
wrong one -- a metric_skill/alert_skill item run against a
visualization-only agent never passes, regardless of question quality.
ChatClient never sent an agentId at all, so there was no way to pick.
- ChatClient gains an `agent_id` param; `create_conversation()` sends
`{"agentId": ...}` only when set -- omitted entirely when None, so
existing behavior (platform's own default-agent resolution) is
unchanged unless the caller opts in.
- Threaded through all 7 run_agentic_*/evaluate_agentic_* pairs, the
agentic-dispatch layer (_dispatch_agentic/run_agentic_items), and the
non-agentic ChatClient construction in cli/main.py.
- New `gd-eval run --agent-id ID` flag (or `GD_EVAL_AGENT_ID` env var,
same precedence convention as --token/GOODDATA_TOKEN).
- README: new flags-table row + a "Targeting a specific AI Hub agent"
section with real usage examples.
- Tests: ChatClient POST-body shape (with/without agent_id), CLI arg
parsing, flag/env-var/unset precedence into the constructed
ChatClient, and _dispatch_agentic threading it to evaluate_agentic_*.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- sse_client.py: reformat create_conversation() to ruff's line-length rule (the PR's own change). - test_cli.py: drop an unused `original_chat_client` local -- pre-existing on master (same line, unrelated to this PR), but ruff check runs whole-file and blocks this PR's lint-and-format-check job since this test function lives in a file the PR also touches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
codecov flagged one uncovered line in _dispatch_agentic's agent_id threading -- the earlier tests only exercised 2 of 7 kind branches. Parametrized test now covers all 7 (vis_agentic, agentic_visualization, agentic_search, agentic_general_question, agentic_guardrail, agentic_conversation, plus the two already covered). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…r-prompt-fidelity fix(gooddata-eval): stop metric-skill simulated user from dropping MAQL clauses
feat(gooddata-eval): support targeting a specific AI Hub agent
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1744 +/- ##
===========================================
+ Coverage 79.50% 79.69% +0.18%
===========================================
Files 272 272
Lines 19019 19024 +5
===========================================
+ Hits 15121 15161 +40
+ Misses 3898 3863 -35 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to 142bcf0 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/32366405936).