examples/01_intro_to_llmstudio_base.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to LLMstudio\n",
    "In this tutorial you'll be base version of LLMstudio.\n",
    "\n",
    "You'll learn:\n",
    "1. How to connect to any provider available (VertexAI, OpenAI, etc.)\n",
    "2. Make sync and async calls both with and without streaming\n",
    "3. See the metrics and logs\n",
    "\n",
    "First things first:\n",
    "* run `pip install llmstudio`\n",
    "* update your .env file with `GOOGLE_API_KEY` or `OPENAI_API_KEY`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llmstudio.providers import LLM\n",
    "\n",
    "# You can set OPENAI_API_KEY and GOOGLE_API_KEY on .env file or pass as api_key on LLM\n",
    "\n",
    "vertexai = LLM(\"vertexai\")\n",
    "\n",
    "openai = LLM(\"openai\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VertexAI\n",
    "### Chat (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am a large language model, and I don't have a name. I'm like a helpful assistant rather than a person. 😊 \n",
      "\n",
      "How can I help you today? \n",
      "\n",
      "{'cost_usd': 0.00038989999999999994,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.10874526841299874,\n",
      " 'latency_s': 1.8914778232574463,\n",
      " 'output_tokens': 37,\n",
      " 'time_to_first_token_s': 1.0876610279083252,\n",
      " 'tokens_per_second': 4.2294971168219355,\n",
      " 'total_tokens': 41}\n"
     ]
    }
   ],
   "source": [
    "response = vertexai.chat(\"What's your name\", model=\"gemini-1.5-pro-latest\")\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Async (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ChatCompletion(id='c29c8269-5494-4b37-a50c-cd9101615dc7', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I don't have a name. I'm a large language model, and I'm not a person.  😊  How can I help you today? \\n\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767411, model='gemini-1.5-pro-latest', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I don't have a name. I'm a large language model, and I'm not a person.  😊  How can I help you today? \\n\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='vertexai', deployment='gemini-1.5-pro-latest', timestamp=1729767411.577035, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 33, 'total_tokens': 37, 'cost_usd': 0.00034789999999999995, 'latency_s': 1.8610751628875732, 'time_to_first_token_s': 1.1106560230255127, 'inter_token_latency_s': 0.10262114661080497, 'tokens_per_second': 4.298590491953858})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "await vertexai.achat(\"What's your name\", model=\"gemini-1.5-pro-latest\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, a vast and enigmatic expanse, beckons with its celestial wonders.  Count\n",
      "\n",
      "less stars, like diamonds scattered across a velvet canvas, illuminate the cosmic tapestry.  Planets, each a unique world, orbit distant suns, while swirling galaxies paint streaks of light across the void.  Mysterious phenomena, such as black holes and\n",
      "\n",
      " nebulae, ignite our imaginations, reminding us of the universe's awe-inspiring power and our own infinitesimal place within it.  Space exploration, a testament to human curiosity, continues to unveil the secrets hidden within this celestial frontier, inspiring generations to come. \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.0013363,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.20518753745339133,\n",
      " 'latency_s': 3.358720064163208,\n",
      " 'output_tokens': 127,\n",
      " 'time_to_first_token_s': 1.0785109996795654,\n",
      " 'tokens_per_second': 3.572789565893662,\n",
      " 'total_tokens': 135}\n"
     ]
    }
   ],
   "source": [
    "response = vertexai.chat(\"Write a paragfraph about space\", model=\"gemini-1.5-pro-latest\", is_stream=True)\n",
    "for i, chunk in enumerate(response):\n",
    "    if i%2==0:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Async (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Space, a vast and enigmatic expanse, stretches beyond the bounds of our planet, teeming\n",
      "\n",
      " with celestial wonders. It is an ethereal tapestry woven with shimmering stars, swirling galaxies, and mysterious nebulae. Planets, both terrestrial and gaseous, orbit distant suns, while asteroids and comets dance through the cosmic ballet. Black holes, with their immense gravitational pull, warp the fabric of spacetime, challenging our understanding of the universe. Space exploration has unveiled breathtaking images and profound knowledge, igniting our curiosity and inspiring generations to reach for the stars. \n",
      "\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.0012523,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.1812559257854115,\n",
      " 'latency_s': 3.075582981109619,\n",
      " 'output_tokens': 119,\n",
      " 'time_to_first_token_s': 1.054703712463379,\n",
      " 'tokens_per_second': 3.901699311546652,\n",
      " 'total_tokens': 127}\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "async for chunk in await vertexai.achat(\"Write a paragfraph about space\", model=\"gemini-1.5-pro-latest\", is_stream=True):\n",
    "    if i%20==2:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n",
    "    i+=1\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenAI\n",
    "### Chat (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I'm OpenAI's GPT-3, a language model AI. I don't have a personal name, but you can just call me GPT-3 or Assistant. How can I assist you today?\n",
      "{'cost_usd': 0.0006650000000000001,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.009538054466247559,\n",
      " 'latency_s': 1.2397558689117432,\n",
      " 'output_tokens': 43,\n",
      " 'time_to_first_token_s': 0.8574087619781494,\n",
      " 'tokens_per_second': 33.07102714987732,\n",
      " 'total_tokens': 47}\n"
     ]
    }
   ],
   "source": [
    "response = openai.chat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Async (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='0126cc99-49c3-4aba-80f9-8f5f682b6737', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I don't have a personal name, but you can call me Assistant or anything else you prefer. How can I help you today?\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767425, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I don't have a personal name, but you can call me Assistant or anything else you prefer. How can I help you today?\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='openai', deployment='gpt-4o-2024-08-06', timestamp=1729767427.5263271, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 27, 'total_tokens': 31, 'cost_usd': 0.00042500000000000003, 'latency_s': 2.210503339767456, 'time_to_first_token_s': 1.6881210803985596, 'inter_token_latency_s': 0.01930811670091417, 'tokens_per_second': 12.666798324288253})\n",
      "I don't have a personal name, but you can call me Assistant or anything else you prefer. How can I help you today?\n",
      "{'cost_usd': 0.00042500000000000003,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.01930811670091417,\n",
      " 'latency_s': 2.210503339767456,\n",
      " 'output_tokens': 27,\n",
      " 'time_to_first_token_s': 1.6881210803985596,\n",
      " 'tokens_per_second': 12.666798324288253,\n",
      " 'total_tokens': 31}\n"
     ]
    }
   ],
   "source": [
    "response = await openai.achat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response)\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, the final frontier, is a vast and enigmatic expanse that stretches far beyond the reach\n",
      "\n",
      " of our imagination. It is a realm of infinite possibilities, filled with countless celestial bodies such as stars\n",
      "\n",
      ", planets, asteroids, and galaxies, each with its own story and significance. The study of\n",
      "\n",
      " space, known as astronomy, has captivated human curiosity for centuries, spurring technological advancements and deepening\n",
      "\n",
      " our understanding of the universe and our place within it. The recent surge in space exploration efforts, driven\n",
      "\n",
      " by both governmental and private organizations, has reignited a sense of wonder and excitement about what lies beyond\n",
      "\n",
      " our planet. As we venture deeper into this cosmic ocean, we continue to uncover the mysteries of black\n",
      "\n",
      " holes, the potential for extraterrestrial life, and the origins of the universe itself, fueling an un\n",
      "\n",
      "ending quest for knowledge and exploration.\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.002575,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.009639137519333891,\n",
      " 'latency_s': 2.020179033279419,\n",
      " 'output_tokens': 169,\n",
      " 'time_to_first_token_s': 0.40947604179382324,\n",
      " 'tokens_per_second': 83.16094624904626,\n",
      " 'total_tokens': 177}\n"
     ]
    }
   ],
   "source": [
    "response = openai.chat(\"Write a paragfraph about space\", model=\"gpt-4o\", is_stream=True)\n",
    "for i, chunk in enumerate(response):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat Async (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, the vast expanse that exists beyond Earth's atmosphere, is a remarkable frontier teeming with\n",
      "\n",
      " mysteries waiting to be unraveled. It stretches billions of light-years in all directions, encompassing planets,\n",
      "\n",
      " stars, galaxies, and untold cosmic phenomena. The vacuum of space is not empty; rather,\n",
      "\n",
      " it is filled with a diverse array of elements, radiation, and the remnants of ancient celestial events.\n",
      "\n",
      " With advancements in technology, we have begun to explore this infinite realm, sending spacecraft and rovers to\n",
      "\n",
      " distant planets and moons, while telescopes capture the light from stars formed billions of years ago. The\n",
      "\n",
      " study of space invites us to ponder profound questions about our origins, the potential for extraterrestrial life,\n",
      "\n",
      " and the ultimate fate of the universe. Each discovery fuels our curiosity, reminding us of the beauty and\n",
      "\n",
      " complexity of the cosmos we inhabit.\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.0001026,\n",
      " 'input_tokens': 8,\n",
      " 'inter_token_latency_s': 0.011185593233850901,\n",
      " 'latency_s': 2.309730052947998,\n",
      " 'output_tokens': 169,\n",
      " 'time_to_first_token_s': 0.4409818649291992,\n",
      " 'tokens_per_second': 72.73577264389624,\n",
      " 'total_tokens': 177}\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "async for chunk in await openai.achat(\"Write a paragfraph about space\", model=\"gpt-4o-mini\", is_stream=True):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if not chunk.metrics:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    else:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n",
    "    i+=1\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}