cookbook/js_tracing_example_vercel_ai_sdk.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "ed99590b",
      "metadata": {},
      "source": [
        "---\n",
        "description: Learn how to use Langfuse for open source observability/tracing of LLM API routes powered by the Vercel AI SDK.\n",
        "category: Integrations\n",
        "---\n",
        "\n",
        "# Cookbook: Vercel AI SDK (JS/TS)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "ae6bb073",
      "metadata": {},
      "source": [
        "This is a cookbook with an end-to-end example on how to use [Langfuse Tracing](https://langfuse.com/docs/tracing) together with the [Vercel AI SDK](https://sdk.vercel.ai/docs).\n",
        "\n",
        "Vercel AI SDK capabilities (from the docs)\n",
        "> - React Server Components API for streaming Generative UI\n",
        "> - SWR-powered React, Svelte, Vue and Solid helpers for streaming text responses and building chat and completion UIs\n",
        "> - First-class support for LangChain and OpenAI, Anthropic, Cohere, Hugging Face, Fireworks and Replicate\n",
        "> - Node.js, Serverless, and Edge Runtime support\n",
        "> - Callbacks for saving completed streaming responses to a database (in the same request)\n",
        "\n",
        "In this end-to-end example, we use the [stream-lifecycle callbacks](https://sdk.vercel.ai/docs/guides/providers/openai#guide-save-to-database-after-completion) to log all LLM calls to Langfuse via the [Langfuse TS SDK](https://langfuse.com/docs/sdk/typescript/guide). It also supports Node.js, Serverless, and Edge Runtimes and intgrates well with Langchain JS ([integration docs](https://langfuse.com/docs/integrations/langchain)).\n",
        "\n",
        "Hint: this is a deno-notebook and uses deno imports."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fd8fc726",
      "metadata": {},
      "source": [
        "## Backend API Route\n",
        "\n",
        "### Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "26910f18",
      "metadata": {},
      "source": [
        "Initialize Langfuse with your API keys from the project settings in the Langfuse UI. As we will use OpenAI LLMs for this example, we also want to configure an OpenAI client."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "id": "d409b2ad",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { Langfuse } from \"npm:langfuse\"\n",
        "import { Configuration, OpenAIApi } from \"npm:openai-edge\";\n",
        "\n",
        "const langfuse = new Langfuse({\n",
        "  publicKey: \"\",\n",
        "  secretKey: \"\",\n",
        "  baseUrl: \"https://cloud.langfuse.com\",\n",
        "});\n",
        "\n",
        "const openAIconfig = new Configuration({\n",
        "  apiKey: \"\",\n",
        "});\n",
        "const openai = new OpenAIApi(openAIconfig);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "21023047-6895-4b9b-9027-a11f2bc8a834",
      "metadata": {},
      "source": [
        "### Cookbook-only: Add prompt to Langfuse\n",
        "\n",
        "We'll also use [Langfuse Prompt Management](https://langfuse.com/docs/prompts) in this example. To be able to subsequently pull a production prompt from Langfuse, we'll quickly push one to Langfuse.\n",
        "\n",
        "If you copy and paste this example, consider creating the prompt one-off or via the Langfuse UI before moving to prod. Alternatively, you can hardcode your prompts and avoid using Langfuse Prompt Management."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "1e37d560-ca38-4daa-a86d-ce1d481a8e9e",
      "metadata": {},
      "outputs": [],
      "source": [
        "await langfuse.createPrompt({\n",
        "    name: \"qa-prompt\",\n",
        "    prompt: \"You are an extremely helpful assistant. Please assume that the person asking you questions needs your support and is totally honest with you.\",\n",
        "    isActive: true // immediately promote to production\n",
        "})"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "94fc8f0e",
      "metadata": {},
      "source": [
        "### API handler\n",
        "\n",
        "The `ai` package provides a number of interfaces and abstractions.\n",
        "\n",
        "In our example, we will use `OpenAIStream` to efficiently process and stream responses from OpenAI's models, and `StreamingTextResponse` to seamlessly deliver these AI-generated responses as HTTP streams to users."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "id": "ae5f4d48",
      "metadata": {},
      "outputs": [],
      "source": [
        "import { OpenAIStream, StreamingTextResponse } from \"npm:ai\";"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d8362c5e-f2b8-4e70-a338-e23ddcd59460",
      "metadata": {},
      "source": [
        "Include the following if you deploy via Vercel:\n",
        "\n",
        "```typescript\n",
        "// select edge rutime for low latency\n",
        "export const runtime = 'edge';\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "56dd778b",
      "metadata": {},
      "source": [
        "Main API handler."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4f06cbfa",
      "metadata": {},
      "outputs": [],
      "source": [
        "/**\n",
        " * Example application simulating a QA chat bot using Langfuse and Vercel AI framework. \n",
        " * \n",
        " * Creates trace, retrieves a pre-saved prompt template, generates answer, and scores generation.\n",
        " * Utilizes OpenAI API for chat completion.\n",
        " *\n",
        " * @param {Request} req - The request object, containing session information, user ID, and messages.\n",
        " * @param {Response} res - The response object used to send back the processed data.\n",
        " * @returns {StreamingTextResponse} - A streamed response containing the output of the OpenAI model.\n",
        " */\n",
        "export default async function handler(req: Request, res: Response) {\n",
        "    // initialize Langfuse \n",
        "    const trace = langfuse.trace({\n",
        "        name: \"QA\",\n",
        "        // Langfuse session tracking: https://langfuse.com/docs/tracing-features/sessions\n",
        "        sessionId: req.sessionId, \n",
        "        // Langfuse user tracking: https://langfuse.com/docs/tracing-features/users\n",
        "        userId: req.userId,\n",
        "        // Make public, so everyone can view it via its URL (for this demo)\n",
        "        public: true\n",
        "    });\n",
        "    \n",
        "    // Format incoming messages for OpenAI API\n",
        "    const messages = req.messages\n",
        "    const openAiMessages = messages.map(({ content, role }) => ({\n",
        "        content,\n",
        "        role: role,\n",
        "    }));\n",
        "    \n",
        "    // get last message\n",
        "    const sanitizedQuery = messages[messages.length - 1].content.trim();\n",
        "\n",
        "    trace.update({\n",
        "        input: sanitizedQuery,\n",
        "    });\n",
        "    \n",
        "    const promptName = req.promptName\n",
        "    \n",
        "    const promptSpan = trace.span({\n",
        "        name: \"fetch-prompt-from-langfuse\",\n",
        "        input: {\n",
        "            promptName,\n",
        "        },\n",
        "    });\n",
        "    \n",
        "    // retrieve Langfuse prompt template with promptName\n",
        "    const prompt = await langfuse.getPrompt(promptName);\n",
        "    \n",
        "    const promptTemplate = prompt.prompt\n",
        "  \n",
        "    promptSpan.end({\n",
        "        output: { \n",
        "            promptTemplate,\n",
        "        },\n",
        "    });\n",
        "    \n",
        "    // merge prompt template and user input\n",
        "    const assembledMessages = [\n",
        "        {\n",
        "            role: \"system\",\n",
        "            content: promptTemplate,\n",
        "        },\n",
        "        ...openAiMessages,\n",
        "    ];\n",
        "      \n",
        "    const generation = trace.generation({\n",
        "        name: \"generation\",\n",
        "        input: assembledMessages,\n",
        "        model: \"gpt-3.5-turbo\",\n",
        "        prompt, // link to prompt version from Langfuse prompt management\n",
        "    });\n",
        "\n",
        "    const response = await openai.createChatCompletion({\n",
        "        model: \"gpt-3.5-turbo\",\n",
        "        stream: true,\n",
        "        messages: assembledMessages,\n",
        "    });\n",
        "    \n",
        "    // Stream the response from OpenAI\n",
        "    const stream = OpenAIStream(response, {\n",
        "        onStart: () => {\n",
        "            // Add completionStartTime timestamp to be able to break down latency\n",
        "            // into delay until first token and the streaming duration\n",
        "            generation.update({\n",
        "                completionStartTime: new Date(),\n",
        "            });\n",
        "        },\n",
        "        onCompletion: async (completion) => {\n",
        "            generation.end({\n",
        "                output: completion,\n",
        "                // Conditionally log a warning state\n",
        "                level: completion.includes(\"I don't know how to help with that\")\n",
        "                    ? \"WARNING\"\n",
        "                    : \"DEFAULT\",\n",
        "                statusMessage: completion.includes(\"I don't know how to help with that\")\n",
        "                    ? \"Refused to answer\"\n",
        "                    : undefined,\n",
        "            });\n",
        "            // Score generation, assume these answers are all correct\n",
        "            if (!completion.includes(\"I don't know how to help with that\")) {\n",
        "                generation.score({\n",
        "                    name: \"quality\",\n",
        "                    value: 1,\n",
        "                    comment: \"Factually correct\",\n",
        "                });\n",
        "            }\n",
        "            trace.update({\n",
        "                output: completion,\n",
        "            });\n",
        "\n",
        "            // Make sure all events are successifully send to Langfuse before the stream terminates.\n",
        "            await langfuse.flushAsync();\n",
        "\n",
        "            // If you run on Vercel, waitUntil will do this in a non-blocking way\n",
        "            // npm i @vercel/functions\n",
        "            // import { waitUntil } from \"@vercel/functions\";\n",
        "            // waitUntil(langfuse.flushAsync())\n",
        "        },\n",
        "    });\n",
        "\n",
        "    // The AI package makes it super easy to add metadata as headers\n",
        "    // It is a bit hacky, but we can e.g. pass the TraceURL to the frontend\n",
        "    return new StreamingTextResponse(stream, {\n",
        "        headers: {\n",
        "            \"X-Langfuse-Trace-Url\": trace.getTraceUrl()\n",
        "        },\n",
        "    });\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "81fd244c-8f32-4a7e-a79c-8e31d5ceb3db",
      "metadata": {},
      "source": [
        "## Our Frontend\n",
        "\n",
        "If you use React, you'll probably want to use the `ai` package's React hooks for state management and to consume the streamed response. Learn more here: https://sdk.vercel.ai/docs/getting-started#wire-up-a-ui\n",
        "\n",
        "\n",
        "To fit this into a notebook, we'll just call the API route directly."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "id": "c5725f67",
      "metadata": {},
      "outputs": [],
      "source": [
        "// sample request to test handler function\n",
        "const mockRequest = {\n",
        "    \"sessionId\": \"testSession\",\n",
        "    \"userId\": \"testUser\",\n",
        "    \"promptName\": \"qa-prompt\",\n",
        "    \"messages\": [\n",
        "        {\n",
        "            \"role\": \"user\",\n",
        "            \"content\": \"What is love?\"\n",
        "        },\n",
        "    ]\n",
        "}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "id": "62be1faa",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Love is a complex and deep emotion that can manifest in various forms such as romantic love, platonic love, familial love, and love for oneself. It often involves feelings of care, affection, empathy, and a strong bond with another person. Love can bring joy, happiness, and fulfillment to our lives, but it can also be challenging and require effort, communication, and understanding to maintain healthy relationships. Overall, love is a fundamental aspect of human experience that can bring meaning and purpose to our lives.\n"
          ]
        }
      ],
      "source": [
        "const response = await handler(mockRequest);\n",
        "const data = await response.text();\n",
        "console.log(data);"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "d27a0cc8",
      "metadata": {},
      "source": [
        "### Explore the trace in the UI\n",
        "\n",
        "Since we passed the trace url to the frontend via the http header and made it public, we cann access it on the Response object."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "id": "311567ba-b49c-45c9-a0d0-aa8a3d929baa",
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "https://cloud.langfuse.com/trace/14cd44b6-1a56-46af-ba85-3fd91bbf9739\n"
          ]
        }
      ],
      "source": [
        "console.log(response.headers.get(\"X-Langfuse-Trace-Url\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "550debb2",
      "metadata": {},
      "source": [
        "![Trace in Langfuse UI](https://langfuse.com/images/cookbook/js_tracing_example_vercel_ai_sdk_trace.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "4d49e6f8-c846-4b4e-bcc4-f1fee3c815b2",
      "metadata": {},
      "source": [
        "PS: As I ran this notebook a couple of times while putting it together, you can find a public [session](https://langfuse.com/docs/tracing-features/sessions) view of me asking the same question `What is love?` [here](https://cloud.langfuse.com/project/cloramnkj0002jz088vzn1ja4/sessions/testSession)."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f04c45b5-3ac9-4727-829e-6798df9cc2af",
      "metadata": {},
      "source": [
        "## Production Demo\n",
        "\n",
        "We also use the Vercel AI SDK to power the public demo ([Docs Q&A Chatbot](https://langfuse.com/demo)). It's open source, see the full backend route here: [`qa-chatbot.ts`](https://github.com/langfuse/langfuse-docs/blob/main/pages/api/qa-chatbot.ts)."
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Deno",
      "language": "typescript",
      "name": "deno"
    },
    "language_info": {
      "file_extension": ".ts",
      "mimetype": "text/x.typescript",
      "name": "typescript",
      "nb_converter": "script",
      "pygments_lexer": "typescript",
      "version": "5.3.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}