docs/source/notebooks/tool_usage/relational_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "60bb467d-861d-4b07-a48d-8e5aa177c969",
   "metadata": {},
   "source": [
    "# Relational Data \n",
    "\n",
    "In this task, an agent is given access to a set of tools that can be used to make queries across 3 relational tables.\n",
    "\n",
    "The tables contain information about users, locations and foods. The agent must answer questions about the data using the provided tools.\n",
    "\n",
    "The underlying data looks like this (showing first 2 records)\n",
    "\n",
    "User data:\n",
    "\n",
    "|  id  |   name   |       email       | location | favorite_color | favorite_foods |\n",
    "| ---- | -------- | ----------------- | -------- | --------------- | --------------- |\n",
    "|  1   |  Alice   | alice@gmail.com  |    1     |      red        |   [1, 2, 3]    |\n",
    "|  21  |   Bob    | bob@hotmail.com  |    2     |     orange      |   [4, 5, 6]    |\n",
    "\n",
    "\n",
    "Location Data:\n",
    "\n",
    "|  id  |    city      |     current_time      |            current_weather               |\n",
    "| ---- | ------------ | --------------------- | --------------------------------------- |\n",
    "|  1   |  New York    | 2023-11-14 10:30 AM  | Partly Cloudy, Temperature: 68°F       |\n",
    "|  2   | Los Angeles  | 2023-11-14 7:45 AM   | Sunny, Temperature: 75°F               |\n",
    "\n",
    "\n",
    "\n",
    "Food data:\n",
    "\n",
    "|  id  |    name    | calories | allergic_ingredients     |\n",
    "| ---- | ---------- | -------- | ------------------------ |\n",
    "|  1   |   Pizza    |   285    |   [\"Gluten\", \"Dairy\"]    |\n",
    "|  2   | Chocolate  |    50    |   [\"Milk\", \"Soy\"]        |\n",
    "\n",
    "\n",
    "The tools allow to look up information based on ids (e.g., `get_user_email` takes a user id and returns the email),\n",
    "and to search (e.g., `find_foods_by_name` takes a food name and returns a list of results).\n",
    "\n",
    "----------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b39159d0-9ea1-414f-a9d8-4a7b22b3d2cc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_benchmarks import registry"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03488ab1-31ed-41c2-8da2-46b02599b181",
   "metadata": {},
   "source": [
    "For this code to work, please configure LangSmith environment variables with your credentials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "60f22779-a948-4833-8e8c-ace9ef17f56f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "task = registry[\"Tool Usage - Relational Data\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "110bdafa-bdab-4194-90c9-46416d14b2f9",
   "metadata": {},
   "source": [
    "## The Environment\n",
    "\n",
    "Let's check the environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "27b6b0fd-639d-43a7-a730-9acdc5b2f102",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[StructuredTool(name='get_user_name', description=\"get_user_name(user_id: int) -> str - Get the name of the user with the given user ID.\\n\\n        Args:\\n            user_id: The user's ID.\\n\\n        Returns:\\n            The user's name.\", args_schema=<class 'pydantic.v1.main.get_user_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.get_user_name at 0x78f30602fec0>),\n",
       " StructuredTool(name='list_user_ids', description='list_user_ids() -> List[str] - List all the user IDs.', args_schema=<class 'pydantic.v1.main.list_user_idsSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.list_user_ids at 0x78f30602fe20>),\n",
       " StructuredTool(name='find_users_by_name', description='find_users_by_name(name: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find users with the given name.\\n\\n        Args:\\n            name: The name to search for.\\n\\n        Returns:\\n            The list of matching users.', args_schema=<class 'pydantic.v1.main.find_users_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_users_by_name at 0x78f306058040>),\n",
       " StructuredTool(name='find_locations_by_name', description='find_locations_by_name(city: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find locations with the given city name.', args_schema=<class 'pydantic.v1.main.find_locations_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_locations_by_name at 0x78f3060580e0>),\n",
       " StructuredTool(name='find_foods_by_name', description='find_foods_by_name(food: str) -> List[langchain_benchmarks.tool_usage.tasks.relational_data.SearchHit] - Find foods with the given name.', args_schema=<class 'pydantic.v1.main.find_foods_by_nameSchema'>, handle_tool_error=True, func=<function get_available_functions.<locals>.find_foods_by_name at 0x78f306058180>)]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env = task.create_environment()\n",
    "env.tools[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7f1c1242-449c-4536-863d-b62bf6d2dff1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Bob'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.tools[0].invoke({\"user_id\": 21})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "854e139b-a120-4012-bdf4-6394e0b1c42d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 2, 'city': 'Los Angeles'},\n",
       " {'id': 1, 'city': 'New York'},\n",
       " {'id': 3, 'city': 'Chicago'},\n",
       " {'id': 4, 'city': 'Houston'},\n",
       " {'id': 5, 'city': 'Miami'}]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "env.tools[3].invoke({\"city\": \"LA\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b462f7b8-fd42-4613-ab5f-5f3cbbc37d28",
   "metadata": {},
   "source": [
    "## Explore the task\n",
    "\n",
    "For evaluation, we need an agent factory that will create a new instance of an agent executor for every evaluation run.\n",
    "\n",
    "We'll use the `StandardAgentFactory` -- look at the `intro` for more information about what it does and/or how to create a custom one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "81c0e4a1-f56e-4117-8804-4161c642b068",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_openai.chat_models import ChatOpenAI\n",
    "\n",
    "from langchain_benchmarks.tool_usage.agents import StandardAgentFactory\n",
    "\n",
    "model = ChatOpenAI(temperature=0)\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", \"{instructions}\"),  # Populated from task.instructions automatically\n",
    "        (\"human\", \"{question}\"),  # Populated from the test data\n",
    "        (\n",
    "            \"placeholder\",\n",
    "            \"{agent_scratchpad}\",\n",
    "        ),  # Work where the agent can do its work (e.g., call multiple tools)\n",
    "    ]\n",
    ")\n",
    "\n",
    "agent_factory = StandardAgentFactory(task, model, prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "382ff2f6-8099-415e-a58c-e659345f52fc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `find_locations_by_name` with `{'city': 'LA'}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3m[{'id': 2, 'city': 'Los Angeles'}, {'id': 1, 'city': 'New York'}, {'id': 3, 'city': 'Chicago'}, {'id': 4, 'city': 'Houston'}, {'id': 5, 'city': 'Miami'}]\u001b[0m\u001b[32;1m\u001b[1;3m\n",
      "Invoking: `get_current_weather_for_location` with `{'location_id': 2}`\n",
      "\n",
      "\n",
      "\u001b[0m\u001b[36;1m\u001b[1;3mSunny, Temperature: 75°F\u001b[0m\u001b[32;1m\u001b[1;3mThe weather in Los Angeles is sunny with a temperature of 75°F.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'question': 'what is the weather in LA',\n",
       " 'output': 'The weather in Los Angeles is sunny with a temperature of 75°F.',\n",
       " 'intermediate_steps': [(ToolAgentAction(tool='find_locations_by_name', tool_input={'city': 'LA'}, log=\"\\nInvoking: `find_locations_by_name` with `{'city': 'LA'}`\\n\\n\\n\", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo', 'function': {'arguments': '{\"city\":\"LA\"}', 'name': 'find_locations_by_name'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-23ccffb0-3b17-46a4-b42e-5eaa3220b211', tool_calls=[{'name': 'find_locations_by_name', 'args': {'city': 'LA'}, 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo'}], tool_call_chunks=[{'name': 'find_locations_by_name', 'args': '{\"city\":\"LA\"}', 'id': 'call_hJrCZgP4eDgaj6s4RtCKXTOo', 'index': 0}])], tool_call_id='call_hJrCZgP4eDgaj6s4RtCKXTOo'),\n",
       "   [{'id': 2, 'city': 'Los Angeles'},\n",
       "    {'id': 1, 'city': 'New York'},\n",
       "    {'id': 3, 'city': 'Chicago'},\n",
       "    {'id': 4, 'city': 'Houston'},\n",
       "    {'id': 5, 'city': 'Miami'}]),\n",
       "  (ToolAgentAction(tool='get_current_weather_for_location', tool_input={'location_id': 2}, log=\"\\nInvoking: `get_current_weather_for_location` with `{'location_id': 2}`\\n\\n\\n\", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'index': 0, 'id': 'call_lopYjo00MF9mZtnHtiisTqyp', 'function': {'arguments': '{\"location_id\":2}', 'name': 'get_current_weather_for_location'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls'}, id='run-9bba5827-d98b-464d-8028-25eb4a05d227', tool_calls=[{'name': 'get_current_weather_for_location', 'args': {'location_id': 2}, 'id': 'call_lopYjo00MF9mZtnHtiisTqyp'}], tool_call_chunks=[{'name': 'get_current_weather_for_location', 'args': '{\"location_id\":2}', 'id': 'call_lopYjo00MF9mZtnHtiisTqyp', 'index': 0}])], tool_call_id='call_lopYjo00MF9mZtnHtiisTqyp'),\n",
       "   'Sunny, Temperature: 75°F')]}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain import globals\n",
    "\n",
    "globals.set_verbose(True)\n",
    "\n",
    "agent = agent_factory()\n",
    "agent.invoke({\"question\": \"what is the weather in LA\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142ac640-3ce0-4f38-89cd-8d24d65997e4",
   "metadata": {},
   "source": [
    "## Benchmarking\n",
    "\n",
    "See `introduction` and `benchmark all` for information on how to run benchmarks. This notebook is just to here to explain and explore the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e49455cc-13c5-4ea6-bb4b-e61c39ea0267",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}