examples/01_intro_to_llmstudio_core.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro to LLMstudio\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import clients\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "import os\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can set OPENAI_API_KEY environment variable, add it to .env, or pass directly as api_key\n",
    "from llmstudio_core.providers import LLMCore as LLM\n",
    "openai = LLM(\"openai\", api_key=os.environ[\"OPENAI_API_KEY\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (non-stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='39195d7f-a24a-407f-bb75-8d424e59081e', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='I’m an AI language model created by OpenAI, and I don’t have a personal name. Feel free to call me whatever you like! How can I assist you today?', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767449, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output='I’m an AI language model created by OpenAI, and I don’t have a personal name. Feel free to call me whatever you like! How can I assist you today?', chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='openai', deployment='gpt-4o-2024-08-06', timestamp=1729767450.487215, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 36, 'total_tokens': 40, 'cost_usd': 0.0005600000000000001, 'latency_s': 0.8399233818054199, 'time_to_first_token_s': 0.41465115547180176, 'inter_token_latency_s': 0.01112962413478542, 'tokens_per_second': 45.24222187780842})\n",
      "I’m an AI language model created by OpenAI, and I don’t have a personal name. Feel free to call me whatever you like! How can I assist you today?\n",
      "{'cost_usd': 0.0005600000000000001,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.01112962413478542,\n",
      " 'latency_s': 0.8399233818054199,\n",
      " 'output_tokens': 36,\n",
      " 'time_to_first_token_s': 0.41465115547180176,\n",
      " 'tokens_per_second': 45.24222187780842,\n",
      " 'total_tokens': 40}\n"
     ]
    }
   ],
   "source": [
    "response = openai.chat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response)\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Async version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='c4740c0d-7552-48b3-a2b4-2de3c7f4de57', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I don't have a personal name, but you can call me Assistant. How can I help you today?\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767452, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I don't have a personal name, but you can call me Assistant. How can I help you today?\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='openai', deployment='gpt-4o-2024-08-06', timestamp=1729767452.419714, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 22, 'total_tokens': 26, 'cost_usd': 0.00035, 'latency_s': 0.6115198135375977, 'time_to_first_token_s': 0.3512728214263916, 'inter_token_latency_s': 0.011792681433937767, 'tokens_per_second': 37.61120979375414})\n",
      "I don't have a personal name, but you can call me Assistant. How can I help you today?\n",
      "{'cost_usd': 0.00035,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.011792681433937767,\n",
      " 'latency_s': 0.6115198135375977,\n",
      " 'output_tokens': 22,\n",
      " 'time_to_first_token_s': 0.3512728214263916,\n",
      " 'tokens_per_second': 37.61120979375414,\n",
      " 'total_tokens': 26}\n"
     ]
    }
   ],
   "source": [
    "response = await openai.achat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response)\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Chat (stream)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Space, the final frontier, is an awe-inspiring expanse that extends far beyond the horizons\n",
      "\n",
      " of our earthly existence. It is a vast, enigmatic void where galaxies twirl gracefully in the cosmic\n",
      "\n",
      " ballet, and stars are born and die in fiery explosions. This infinite cosmos is dotted with fascinating celestial\n",
      "\n",
      " bodies, from barren yet intriguing planets and moons to swirling nebulas rich in color and complexity. Space\n",
      "\n",
      " is not just a realm of mystery for scientists and astronomers; it beckons humanity with possibilities and\n",
      "\n",
      " challenges, urging us to explore and understand the universe’s origins, the nature of time and space itself\n",
      "\n",
      ", and our place within this grand tapestry. Our ventures into space have broadened our horizons, sp\n",
      "\n",
      "urring technological advancements and uniting nations in the shared quest for knowledge. As we continue to push the\n",
      "\n",
      " boundaries of space exploration, we carry the hope of discovering new worlds, understanding the cosmos more profoundly,\n",
      "\n",
      " and perhaps finding hints of life beyond our pale blue dot.\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 0.00295,\n",
      " 'input_tokens': 5,\n",
      " 'inter_token_latency_s': 0.011840422948201498,\n",
      " 'latency_s': 2.7758469581604004,\n",
      " 'output_tokens': 195,\n",
      " 'time_to_first_token_s': 0.5021190643310547,\n",
      " 'tokens_per_second': 69.52832879803442,\n",
      " 'total_tokens': 200}\n"
     ]
    }
   ],
   "source": [
    "response = openai.chat(\"Write a paragraph about space\", model=\"gpt-4o\", is_stream=True)\n",
    "for i, chunk in enumerate(response):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if chunk.chat_output_stream:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    elif chunk.metrics:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Async version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Hi there! How can I assist you today?\n",
      "\n",
      "## Metrics:\n",
      "{'cost_usd': 6.45e-06,\n",
      " 'input_tokens': 3,\n",
      " 'inter_token_latency_s': 0.010372812097722834,\n",
      " 'latency_s': 0.5299859046936035,\n",
      " 'output_tokens': 10,\n",
      " 'time_to_first_token_s': 0.41546201705932617,\n",
      " 'tokens_per_second': 22.642111599057458,\n",
      " 'total_tokens': 13}\n"
     ]
    }
   ],
   "source": [
    "i=0\n",
    "async for chunk in await openai.achat(\"Say hi back\", model=\"gpt-4o-mini\", is_stream=True):\n",
    "    if i%20==0:\n",
    "        print(\"\\n\")\n",
    "    if chunk.chat_output_stream:\n",
    "        print(chunk.chat_output_stream, end=\"\", flush=True)\n",
    "    elif chunk.metrics:\n",
    "        print(\"\\n\\n## Metrics:\")\n",
    "        pprint(chunk.metrics)\n",
    "    i+=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AzureOpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = LLM(provider=\"azure\", \n",
    "          api_key=os.environ[\"AZURE_API_KEY\"], \n",
    "          api_version=os.environ[\"AZURE_API_VERSION\"],\n",
    "          api_endpoint=os.environ[\"AZURE_API_ENDPOINT\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='8b9fd3d7-839e-4ad6-be59-6242b8404eee', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I don't have a personal name, but you can call me Assistant. How can I help you today?\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767460, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I don't have a personal name, but you can call me Assistant. How can I help you today?\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='azure', deployment='gpt-4o-2024-05-13', timestamp=1729767461.879921, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 22, 'total_tokens': 26, 'cost_usd': 0.00035, 'latency_s': 1.7873499393463135, 'time_to_first_token_s': 0.6913847923278809, 'inter_token_latency_s': 0.04434918273578991, 'tokens_per_second': 12.868213153833645})\n",
      "I don't have a personal name, but you can call me Assistant. How can I help you today?\n",
      "{'cost_usd': 0.00035,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.04434918273578991,\n",
      " 'latency_s': 1.7873499393463135,\n",
      " 'output_tokens': 22,\n",
      " 'time_to_first_token_s': 0.6913847923278809,\n",
      " 'tokens_per_second': 12.868213153833645,\n",
      " 'total_tokens': 26}\n"
     ]
    }
   ],
   "source": [
    "response = llm.chat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response)\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ChatCompletion(id='85ebdb8b-439b-4736-9b22-478f94696f7a', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content=\"I don't have a name, but you can call me Assistant! How can I help you today?\", refusal=None, role='assistant', audio=None, function_call=None, tool_calls=None))], created=1729767462, model='gpt-4o', object='chat.completion', service_tier=None, system_fingerprint=None, usage=None, chat_input=\"What's your name\", chat_output=\"I don't have a name, but you can call me Assistant! How can I help you today?\", chat_output_stream='', context=[{'role': 'user', 'content': \"What's your name\"}], provider='azure', deployment='gpt-4o-2024-05-13', timestamp=1729767463.2252412, parameters={}, metrics={'input_tokens': 4, 'output_tokens': 21, 'total_tokens': 25, 'cost_usd': 0.000335, 'latency_s': 0.6665310859680176, 'time_to_first_token_s': 0.25866127014160156, 'inter_token_latency_s': 0.013548714773995536, 'tokens_per_second': 33.006712609733604})\n",
      "I don't have a name, but you can call me Assistant! How can I help you today?\n",
      "{'cost_usd': 0.000335,\n",
      " 'input_tokens': 4,\n",
      " 'inter_token_latency_s': 0.013548714773995536,\n",
      " 'latency_s': 0.6665310859680176,\n",
      " 'output_tokens': 21,\n",
      " 'time_to_first_token_s': 0.25866127014160156,\n",
      " 'tokens_per_second': 33.006712609733604,\n",
      " 'total_tokens': 25}\n"
     ]
    }
   ],
   "source": [
    "response = await llm.achat(\"What's your name\", model=\"gpt-4o\")\n",
    "print(response)\n",
    "print(response.chat_output)\n",
    "pprint(response.metrics)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}