Split out of #1762 discussion (agreed there as separable).
_set_skills_declarations() / _final_skill_declaration() in packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py derive the active skill set from the set_skills tool call's arguments — i.e. what the agent asked for. The same is true of _check_visualization_skill_activated in core/evaluators/visualization.py.
That is not always what became active. The service:
- drops requested names it does not recognise, and
- pulls in skills declared as dependencies of the ones requested.
The tool's own result echoes back the authoritative post-replacement set, so reading the result would be strictly truer to what actually became active than re-deriving it from the request.
Consequence today: a fixture requesting a misspelled or retired skill name is credited with activating it, and a skill activated only as a dependency is not credited at all — so both skill_routing and full_skill_coverage can be wrong in either direction.
Left out of #1762 because it is a behaviour change to how activation is measured, independent of the replace-semantics fix that PR is about, and it needs its own fixtures.
Scope:
- Parse the
set_skills tool result and prefer it over the arguments, falling back to arguments when the result is absent or unparseable (older captured traces, and any trace where the call errored).
- Same treatment in
core/evaluators/visualization.py.
- Tests covering: unrecognised name requested, dependency pulled in implicitly, and result missing (fallback path).
Split out of #1762 discussion (agreed there as separable).
_set_skills_declarations()/_final_skill_declaration()inpackages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.pyderive the active skill set from theset_skillstool call's arguments — i.e. what the agent asked for. The same is true of_check_visualization_skill_activatedincore/evaluators/visualization.py.That is not always what became active. The service:
The tool's own result echoes back the authoritative post-replacement set, so reading the result would be strictly truer to what actually became active than re-deriving it from the request.
Consequence today: a fixture requesting a misspelled or retired skill name is credited with activating it, and a skill activated only as a dependency is not credited at all — so both
skill_routingandfull_skill_coveragecan be wrong in either direction.Left out of #1762 because it is a behaviour change to how activation is measured, independent of the replace-semantics fix that PR is about, and it needs its own fixtures.
Scope:
set_skillstool result and prefer it over the arguments, falling back to arguments when the result is absent or unparseable (older captured traces, and any trace where the call errored).core/evaluators/visualization.py.