examples/01_intro_to_llmstudio_with_proxy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to LLMstudio - proxy\n",
    "\n",
    "This tutorial serves as an introduction to LLMstudio, a platform designed to facilitate interactions with large language models (LLMs) through a proxy server. By following this guide, users will learn how to set up and run a proxy server that acts as an intermediary between the client and the LLM provider, ensuring seamless communication and data handling. The notebook demonstrates the process of initializing the server, configuring the proxy, and making requests to the LLM, providing a comprehensive overview of the system's capabilities.\n",
    "\n",
    "Additionally, the tutorial highlights the integration of tracking features within LLMstudio, allowing users to monitor and log interactions with the LLMs. This is particularly useful for analyzing performance metrics, such as latency and token usage, which can help in optimizing the model's responses and understanding its behavior in different scenarios. By the end of this tutorial, users will have a solid foundation in setting up and utilizing LLMstudio for efficient and effective LLM interactions.\n",
    "\n",
    "You'll learn:\n",
    "1. How to start a local proxy server\n",
    "1. How to connect to any provider available (VertexAI, OpenAI, etc.)\n",
    "2. Make sync and async calls both with and without streaming\n",
    "3. See the save logs\n",
    "\n",
    "First things first:\n",
    "* run `pip install llmstudio[proxy]`\n",
    "* update your .env file with `GOOGLE_API_KEY` or `OPENAI_API_KEY`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## start proxy server\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running LLMstudio Proxy on http://0.0.0.0:8001 \n"
     ]
    }
   ],
   "source": [
    "from llmstudio.server import start_servers\n",
    "\n",
    "# default port is 50001. set the environment varible to specify which host and port; LLMSTUDIO_ENGINE_HOST, LLMSTUDIO_ENGINE_PORT\n",
    "start_servers(proxy=True, tracker=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected to LLMStudio Proxy @ 0.0.0.0:8001\n"
     ]
    }
   ],
   "source": [
    "from llmstudio_proxy.provider import LLMProxyProvider as LLM\n",
    "from llmstudio_proxy.provider import ProxyConfig\n",
    "\n",
    "llm = LLM(provider=\"openai\", \n",
    "          proxy_config=ProxyConfig(host=\"0.0.0.0\", port=\"8001\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = llm.chat(\"olá\", model=\"gpt-4o\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Olá! Como posso ajudá-lo hoje?',\n",
       " {'input_tokens': 2,\n",
       "  'output_tokens': 11,\n",
       "  'total_tokens': 13,\n",
       "  'cost_usd': 0.000175,\n",
       "  'latency_s': 0.719973087310791,\n",
       "  'time_to_first_token_s': 0.547133207321167,\n",
       "  'inter_token_latency_s': 0.018544991811116535,\n",
       "  'tokens_per_second': 13.889408057392146})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result.chat_output, result.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the LLMStudio SDK entrypoint\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llmstudio.providers import LLM\n",
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "host='0.0.0.0' port='8001' url=None username=None password=None\n",
      "Connected to LLMStudio Proxy @ 0.0.0.0:8001\n"
     ]
    }
   ],
   "source": [
    "# You can set OPENAI_API_KEY and ANTHROPIC_API_KEY on .env file\n",
    "from llmstudio_proxy.provider import ProxyConfig\n",
    "proxy = ProxyConfig(host=\"0.0.0.0\", port=\"8001\")\n",
    "print(proxy)\n",
    "\n",
    "openai = LLM(\"openai\", proxy_config=proxy)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<llmstudio_proxy.provider.LLMProxyProvider at 0x1167c0530>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "openai._provider\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='38cd37e1-6eec-43df-a75c-ab3445a78f1c', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I'm an AI developed by OpenAI, and I don't have a personal name. You can call me Assistant if you'd like!\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767674, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I'm an AI developed by OpenAI, and I don't have a personal name. You can call me Assistant if you'd like!\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='openai', deployment='gpt-4o-2024-08-06', timestamp=1729767675.315573, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 28, 'total_tokens': 32, 'cost_usd': 0.00044, 'latency_s': 1.1188991069793701, 'time_to_first_token_s': 0.7827062606811523, 'inter_token_latency_s': 0.012905460137587327, 'tokens_per_second': 24.130862051441262})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "openai.chat(\"What's your name\", model=\"gpt-4o\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Async version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='cbe07ad8-04b6-4423-986d-ae2be63ef0a8', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='I’m called ChatGPT. How can I assist you today?', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767676, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output='I’m called ChatGPT. How can I assist you today?', chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='openai', deployment='gpt-4o-2024-08-06', timestamp=1729767677.5680408, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 14, 'total_tokens': 18, 'cost_usd': 0.00023, 'latency_s': 0.8708088397979736, 'time_to_first_token_s': 0.5257279872894287, 'inter_token_latency_s': 0.024605206080845425, 'tokens_per_second': 17.225364872823267})\n"
     ]
    }
   ],
   "source": [
    "result = await openai.achat(\"What's your name\", model=\"gpt-4o\")\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, the vast expanse that stretches beyond the confines of our planet, is a realm of\n",
      "\n",
      " wonder and mystery that has captivated human imagination for centuries. It is a place where stars are born,\n",
      "\n",
      " galaxies collide, and black holes exert their powerful pull, shaping the cosmos in ways we are just beginning\n",
      "\n",
      " to understand. Space exploration has advanced dramatically since the mid-20th century, with missions that have\n",
      "\n",
      " taken us to the Moon, Mars, and beyond, providing glimpses of distant worlds and enhancing our\n",
      "\n",
      " understanding of the universe. This infinite frontier challenges our scientific knowledge and technological prowess, driving innovation and inspiring\n",
      "\n",
      " a sense of awe as we ponder the possibilities of life beyond Earth and the origins of the universe itself\n",
      "\n",
      ". As we continue our journey into space, we are reminded of the boundless potential that lies beyond\n",
      "\n",
      " our sky, waiting to be discovered.\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.00256,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.030803951479139783,\n",
      " 'latency_s': 5.725583076477051,\n",
      " 'output_tokens': 168,\n",
      " 'time_to_first_token_s': 0.5494673252105713,\n",
      " 'tokens_per_second': 29.516644460949056,\n",
      " 'total_tokens': 176}\n"
     ]
    }
   ],
   "source": [
    "response = openai.chat(\"Write a paragfraph about space\", model=\"gpt-4o\", is_stream=True)\n",
    "for i, chunk in enumerate(response):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Async version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, the vast and enigmatic expanse that stretches beyond our atmosphere, is a realm of infinite\n",
      "\n",
      " wonder and discovery. It is home to billions of galaxies, each containing stars, planets, and potentially\n",
      "\n",
      " life, all governed by the fundamental laws of physics. The beauty of space captivates our imagination,\n",
      "\n",
      " from the shimmering glow of distant stars to the mesmerizing swirl of galaxies. It serves not only as a\n",
      "\n",
      " backdrop for our planet but also as the stage for humanity's quest for knowledge, as we launch missions\n",
      "\n",
      " to explore celestial bodies, probe the mysteries of black holes, and decode the origins of the universe.\n",
      "\n",
      " As we look up at the night sky, we are reminded of our place in the cosmos, spar\n",
      "\n",
      "king curiosity and a deep desire to understand the forces that shape our existence.\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 9.54e-05,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.011587064496932491,\n",
      " 'latency_s': 2.327955961227417,\n",
      " 'output_tokens': 157,\n",
      " 'time_to_first_token_s': 0.5314009189605713,\n",
      " 'tokens_per_second': 67.01157693625306,\n",
      " 'total_tokens': 165}\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "async for chunk in await openai.achat(\"Write a paragfraph about space\", model=\"gpt-4o-mini\", is_stream=True):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n",
    "    i+=1\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}