v1.13.0

Breaking Changes

Note that 2 methods from the Sessions class have been deprecated.

• Sessions.preset_parameters
• Sessions.run_conversation

For each of these methods, you can use Sessions.detect_intent instead, which is fully backwards compatible.

New Features

Agent Tasks Generator

This is a specialized tool that allows a user to evaluate any arbitrary agent to determine what the Agent is capable of accomplishing from a task perspective.
This is a pre-release feature that will be accompanied by more automated testing features in the future.

For now, you can use this as a way to analyze arbitrary Agents to see if they are set up to perform the tasks you believe you configured them to do.

from dfcx_scrapi.tools.agent_task_generator import AgentTaskGenerator

atg = AgentTaskGenerator(agent_id=agent_id)
atg.get_agent_tasks()

Output

{'tasks': [{'name': 'Greeting and Intent Understanding',
   'description': 'The agent greets the user and attempts to understand their intent. It can provide basic information like translations or virtual money, and direct the user to appropriate tools or flows based on their request.'},
  {'name': 'Product and Company Information Retrieval',
   'description': 'The agent can access a data store containing information from the YETI website to answer user queries about YETI products and the company.'},
  {'name': 'Trip Planning Assistance',
   'description': 'The agent can collect basic information from the user to assist with trip planning, including destination, travel dates, and preferences. It can then pass this information to a separate flow for further processing.'}]}

Evaluation Dataset from Conversation History

You can now quickly create an Evaluation dataset format using pre-selected conversations from the Conversation History in your Agent.
Simply select the list of conversation_ids that you want, and pass that to the Evals.create_dataset_from_conv_ids method, which will provide a Pandas Dataframe in return.

This can be saved as a CSV, Google Sheet, or used locally to run Evals on your Agent.

from dfcx_scrapi.core.conversation_history import ConversationHistory
from dfcx_scrapi.tools.evaluations import Evaluations

ch = ConversationHistory()
evals = Evaluations(agent_id=agent_id)

all_convos = ch.list_conversations(agent_id)
convo_ids = [convo.name for convo in all_convos[:5]]
evals.create_dataset_from_conv_ids(convo_ids)

Output

eval_id	action_id	action_type	action_input	action_input_parameters	tool_action	notes
0	001	1	User Utterance	what items do you have for dogs?
1	001	2	Tool Invocation	yeti-website	{'requestBody': {'query': 'what items do you h...	yeti-website
2	001	3	Agent Response	YETI offers dog bowls and dog beds. The Boomer...
3	002	1	User Utterance	who is the ceo?
4	002	2	Tool Invocation	yeti-website	{'requestBody': {'query': 'who is the ceo?'}}	yeti-website
5	002	3	Agent Response	The CEO of YETI is Matt Reintjes.
6	003	1	User Utterance	I want to speak to an operator
7	003	2	Agent Response	Just a moment while I connect you...
8	004	1	User Utterance	where is yeti hq at?
9	004	2	Tool Invocation	yeti-website	{'requestBody': {'query': 'where is yeti hq at...	yeti-website
10	004	3	Agent Response	YETI's headquarters is located in Austin, Texa...
11	005	1	User Utterance	what is the smallest cup I can buy?
12	005	2	Tool Invocation	yeti-website	{'requestBody': {'query': 'what is the smalles...	yeti-website
13	005	3	Agent Response	The smallest cup you can buy is the 4oz cup. I...

CICD Workflow Example

We've added an example CICD workflow for anyone that is curious to see how a CICD workflow could be set up using SCRAPI.
Fair warning, it's very involved! 😄
However, it provides some good pointers on how you can set up these types of complex pipelines using this library.

Enhancements

• Added support for environment_id when calling Sessions.build_session_id, which allows you to now use Sessions.detect_intent with a session_id that includes an Environment
• Added language_code support throughout Evaluations class
• Added support for setting BigQuery logging and interaction settings
• Added support for new lint rules in ruff
• Updated some out of date Example notebooks and fixed broken links
• Added support for Session Parameters in Evaluations

Bug Fix

• Fixed several issues in Evaluations where dataframe prep and parsing where failing

What's Changed

• Feat/agent tasks generator by @kmaphoenix in #259
• feat: Implement Python requests-based get/set for BigQuery Interaction Logging Settings by @justin-oos in #258
• Feat/bq update sdk by @kmaphoenix in #263
• Fill Golden Template from Conv_Ids by @gmchueh in #261
• Fix/update lint rules by @kmaphoenix in #264
• Clean example notebooks (bot builder / vertex ai / google sheets) by @ethanknights in #256
• Adding session parameters to evaluations function by @AAMEHROTRA1230 in #262
• Feature/dfcxcicd by @sridharvikram in #239
• fix: add lang_code support for DataLoader by @kmaphoenix in #265
• fix/evaluations multiple tool pairing and empty utterance pairing by @gmchueh in #267
• Update google doc reference in nlu_evaluation_testing.ipynb by @ethanknights in #266
• Fix/session id with env by @kmaphoenix in #271

New Contributors

• @justin-oos made their first contribution in #258
• @gmchueh made their first contribution in #261
• @ethanknights made their first contribution in #256
• @AAMEHROTRA1230 made their first contribution in #262
• @sridharvikram made their first contribution in #239

Full Changelog: 1.12.5...1.13.0