cookbook/integration_azure_openai_langchain.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KlceIPalN3QR"
   },
   "source": [
    "---\n",
    "description: This cookbook demonstate use of Langfuse with Azure OpenAI and Langchain for prompt versioning and evaluations.\n",
    "category: Integrations\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mqBspBzuRk9C"
   },
   "source": [
    "# Langfuse Tracing and Prompt Management for Azure OpenAI and Langchain\n",
    "\n",
    "This cookbook demonstate use of Langfuse with Azure OpenAI and Langchain for prompt versioning and evaluations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AbSpd5EiZouE"
   },
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YNyU6IzCZouE"
   },
   "outputs": [],
   "source": [
    "%pip install --quiet langfuse langchain langchain-openai --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# get keys for your project from https://cloud.langfuse.com\n",
    "os.environ[\"LANGFUSE_PUBLIC_KEY\"] = \"pk-lf-***\"\n",
    "os.environ[\"LANGFUSE_SECRET_KEY\"] = \"sk-lf-***\"\n",
    "os.environ[\"LANGFUSE_HOST\"] = \"https://cloud.langfuse.com\" # for EU data region\n",
    "# os.environ[\"LANGFUSE_HOST\"] = \"https://us.cloud.langfuse.com\" # for US data region\n",
    "\n",
    "# your azure openai configuration\n",
    "os.environ[\"AZURE_OPENAI_ENDPOINT\"] = \"your Azure OpenAI endpoint\"\n",
    "os.environ[\"AZURE_OPENAI_API_KEY\"] = \"your Azure OpenAI API key\"\n",
    "os.environ[\"OPENAI_API_TYPE\"] = \"azure\"\n",
    "os.environ[\"OPENAI_API_VERSION\"] = \"2023-09-01-preview\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the native Langfuse intgeration for Langchain. Learn more it in the [documentation](https://langfuse.com/docs/integrations/langchain)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "divRadPqZouF"
   },
   "outputs": [],
   "source": [
    "from langfuse.callback import CallbackHandler\n",
    "\n",
    "langfuse_handler = CallbackHandler()\n",
    "\n",
    "# optional, verify your Langfuse credentials\n",
    "langfuse_handler.auth_check()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FzsO2Ci4EjJS"
   },
   "source": [
    "Langchain imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L9u5QMVCbPtR"
   },
   "outputs": [],
   "source": [
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.prompts.chat import (\n",
    "    ChatPromptTemplate,\n",
    "    SystemMessagePromptTemplate,\n",
    "    HumanMessagePromptTemplate,\n",
    ")\n",
    "from langchain.schema import HumanMessage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langfuse.callback import CallbackHandler\n",
    "\n",
    "langfuse_handler = CallbackHandler()\n",
    "\n",
    "prompt = ChatPromptTemplate.from_template(\"what is the city {person} is from?\")\n",
    "model = AzureChatOpenAI(\n",
    "    deployment_name=\"gpt-35-turbo\",\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    ")\n",
    "chain = prompt | model\n",
    "\n",
    "chain.invoke({\"person\": \"Satya Nadella\"}, config={\"callbacks\":[langfuse_handler]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**✨ Done. Go to the Langfuse Dashboard to explore the trace of this run.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KAW-Gt4mN3QV"
   },
   "source": [
    "## Example using Langfuse Prompt Management and Langchain\n",
    "\n",
    "Learn more about Langfuse Prompt Management in the [docs](https://langfuse.com/docs/prompts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize the Langfuse Client\n",
    "from langfuse import Langfuse\n",
    "langfuse = Langfuse()\n",
    "\n",
    "template = \"\"\"\n",
    "You are an AI assistant travel assistant that provides vacation recommendations to users. \n",
    "You should also be able to provide information about the weather, local customs, and travel restrictions. \n",
    "\"\"\"\n",
    "\n",
    "# Push the prompt to Langfuse and immediately promote it to production\n",
    "langfuse.create_prompt(\n",
    "    name=\"travel_consultant\",\n",
    "    prompt=template,\n",
    "    is_active=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In your production environment, you can then fetch the production version of the prompt. The Langfuse client caches the prompt to improve performance. You can configure this behavior via a custom TTL or disable it completely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the prompt from Langfuse, cache it for 5 minutes\n",
    "langfuse_prompt = langfuse.get_prompt(\"travel_consultant\", cache_ttl_seconds=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do not use the native Langfuse `prompt.compile()` but use the raw `prompt.prompt` as Langchain will insert the prompt variables (if any)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_message_prompt = SystemMessagePromptTemplate.from_template(langfuse_prompt.prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = AzureChatOpenAI(\n",
    "    deployment_name=\"gpt-35-turbo\",\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    ")\n",
    "\n",
    "human_message_prompt = HumanMessagePromptTemplate.from_template(\"{text}\")\n",
    "chat_prompt = ChatPromptTemplate.from_messages(\n",
    "    [system_message_prompt, human_message_prompt]\n",
    ")\n",
    "chain = LLMChain(llm=llm, prompt=chat_prompt)\n",
    "result = chain.run(\n",
    "    f\"Where should I go on vaction in Decemember for warm weather and beaches?\",\n",
    "    callbacks=[langfuse_handler],\n",
    ")\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multiple Langchain runs in same Langfuse trace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Langchain setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_openai import AzureChatOpenAI\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.schema import StrOutputParser\n",
    "from operator import itemgetter\n",
    "\n",
    "prompt1 = ChatPromptTemplate.from_template(\n",
    "    \"What {type} is easiest to learn but hardest to master? Give a step by step approach of your thoughts, ending in your answer\"\n",
    ")\n",
    "prompt2 = ChatPromptTemplate.from_template(\n",
    "    \"How {type} can be learned in 21 days? respond in {language}\"\n",
    ")\n",
    "model = AzureChatOpenAI(\n",
    "    deployment_name=\"gpt-35-turbo\",\n",
    "    model_name=\"gpt-3.5-turbo\",\n",
    ")\n",
    "chain1 = prompt1 | model | StrOutputParser()\n",
    "chain2 = (\n",
    "    {\"type\": chain1, \"language\": itemgetter(\"language\")}\n",
    "    | prompt2\n",
    "    | model\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the chain multiple times within the same Langfuse trace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create trace using Langfuse Client\n",
    "langfuse = Langfuse()\n",
    "trace = langfuse.trace(name=\"chain_of_thought_example\", user_id=\"user-1234\")\n",
    "\n",
    "# Create a handler scoped to this trace\n",
    "langfuse_handler = trace.get_langchain_handler()\n",
    "\n",
    "# First run\n",
    "chain2.invoke(\n",
    "    {\"type\": \"business\", \"language\": \"german\"}, config={\"callbacks\": [langfuse_handler]}\n",
    ")\n",
    "\n",
    "# Second run\n",
    "chain2.invoke(\n",
    "    {\"type\": \"business\", \"language\": \"english\"}, config={\"callbacks\": [langfuse_handler]}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "AxQlUOmVPEwz"
   },
   "source": [
    "## Adding scores\n",
    "\n",
    "When evaluating traces of your LLM application in Langfuse, you need to add [scores](https://langfuse.com/docs/scores) to the trace. For simplicity, we'll add a mocked score. Check out the docs for more information on complex score types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the trace_id. We use the previous run where we created the trace using `langfuse.trace()`. You can also get the trace_id via `langfuse_handler.get_trace_id()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trace_id = trace.id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PudCopwEPFgh"
   },
   "outputs": [],
   "source": [
    "# Add score to the trace via the Langfuse Python Client\n",
    "langfuse = Langfuse()\n",
    "\n",
    "trace = langfuse.score(\n",
    "    trace_id=trace_id,\n",
    "    name=\"user-explicit-feedback\",\n",
    "    value=1,\n",
    "    comment=\"I like how personalized the response is\",\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}