Skip to content

gooddata-eval: read set_skills' result rather than its arguments to determine active skills #1779

Description

@Tomkess

Split out of #1762 discussion (agreed there as separable).

_set_skills_declarations() / _final_skill_declaration() in packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py derive the active skill set from the set_skills tool call's arguments — i.e. what the agent asked for. The same is true of _check_visualization_skill_activated in core/evaluators/visualization.py.

That is not always what became active. The service:

  • drops requested names it does not recognise, and
  • pulls in skills declared as dependencies of the ones requested.

The tool's own result echoes back the authoritative post-replacement set, so reading the result would be strictly truer to what actually became active than re-deriving it from the request.

Consequence today: a fixture requesting a misspelled or retired skill name is credited with activating it, and a skill activated only as a dependency is not credited at all — so both skill_routing and full_skill_coverage can be wrong in either direction.

Left out of #1762 because it is a behaviour change to how activation is measured, independent of the replace-semantics fix that PR is about, and it needs its own fixtures.

Scope:

  • Parse the set_skills tool result and prefer it over the arguments, falling back to arguments when the result is absent or unparseable (older captured traces, and any trace where the call errored).
  • Same treatment in core/evaluators/visualization.py.
  • Tests covering: unrecognised name requested, dependency pulled in implicitly, and result missing (fallback path).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions