02-open-source/homework 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5c4f92ae-fa2e-4a51-b542-92c98ca5e1db",
   "metadata": {},
   "source": [
    "# Questions List (Answers follow after)\n",
    "\n",
    "[Source](https://github.com/DataTalksClub/llm-zoomcamp/blob/main/cohorts/2024/02-open-source/homework.md)\n",
    "\n",
    "In this homework, we'll experiment more with Ollama\n",
    "\n",
    "> It's possible that your answers won't match exactly. If it's the case, select the closest one.\n",
    "\n",
    "## Q1. Running Ollama with Docker\n",
    "\n",
    "Let's run ollama with Docker. We will need to execute the \n",
    "same command as in the lectures:\n",
    "\n",
    "```bash\n",
    "docker run -it \\\n",
    "    --rm \\\n",
    "    -v ollama:/root/.ollama \\\n",
    "    -p 11434:11434 \\\n",
    "    --name ollama \\\n",
    "    ollama/ollama\n",
    "```\n",
    "\n",
    "What's the version of ollama client? \n",
    "\n",
    "To find out, enter the container and execute `ollama` with the `-v` flag.\n",
    "\n",
    "\n",
    "## Q2. Downloading an LLM \n",
    "\n",
    "We will donwload a smaller LLM - gemma:2b. \n",
    "\n",
    "Again let's enter the container and pull the model:\n",
    "\n",
    "```bash\n",
    "ollama pull gemma:2b\n",
    "```\n",
    "\n",
    "In docker, it saved the results into `/root/.ollama`\n",
    "\n",
    "We're interested in the metadata about this model. You can find\n",
    "it in `models/manifests/registry.ollama.ai/library`\n",
    "\n",
    "What's the content of the file related to gemma?\n",
    "\n",
    "## Q3. Running the LLM\n",
    "\n",
    "Test the following prompt: \"10 * 10\". What's the answer?\n",
    "\n",
    "## Q4. Donwloading the weights \n",
    "\n",
    "We don't want to pull the weights every time we run\n",
    "a docker container. Let's do it once and have them available\n",
    "every time we start a container.\n",
    "\n",
    "First, we will need to change how we run the container.\n",
    "\n",
    "Instead of mapping the `/root/.ollama` folder to a named volume,\n",
    "let's map it to a local directory:\n",
    "\n",
    "```bash\n",
    "mkdir ollama_files\n",
    "\n",
    "docker run -it \\\n",
    "    --rm \\\n",
    "    -v ./ollama_files:/root/.ollama \\\n",
    "    -p 11434:11434 \\\n",
    "    --name ollama \\\n",
    "    ollama/ollama\n",
    "```\n",
    "\n",
    "Now pull the model:\n",
    "\n",
    "```bash\n",
    "docker exec -it ollama ollama pull gemma:2b \n",
    "```\n",
    "\n",
    "What's the size of the `ollama_files/models` folder? \n",
    "\n",
    "* 0.6G\n",
    "* 1.2G\n",
    "* 1.7G\n",
    "* 2.2G\n",
    "\n",
    "Hint: on linux, you can use `du -h` for that.\n",
    "\n",
    "## Q5. Adding the weights \n",
    "\n",
    "Let's now stop the container and add the weights \n",
    "to a new image\n",
    "\n",
    "For that, let's create a `Dockerfile`:\n",
    "\n",
    "```dockerfile\n",
    "FROM ollama/ollama\n",
    "\n",
    "COPY ...\n",
    "```\n",
    "\n",
    "What do you put after `COPY`?\n",
    "\n",
    "## Q6. Serving it \n",
    "\n",
    "Let's build it:\n",
    "\n",
    "```bash\n",
    "docker build -t ollama-gemma2b .\n",
    "```\n",
    "\n",
    "And run it:\n",
    "\n",
    "```bash\n",
    "docker run -it --rm -p 11434:11434 ollama-gemma2b\n",
    "```\n",
    "\n",
    "We can connect to it using the OpenAI client\n",
    "\n",
    "Let's test it with the following prompt:\n",
    "\n",
    "```python\n",
    "prompt = \"What's the formula for energy?\"\n",
    "```\n",
    "\n",
    "Also, to make results reproducible, set the `temperature` parameter to 0:\n",
    "\n",
    "```bash\n",
    "response = client.chat.completions.create(\n",
    "    #...\n",
    "    temperature=0.0\n",
    ")\n",
    "```\n",
    "\n",
    "How many completion tokens did you get in response?\n",
    "\n",
    "* 304\n",
    "* 604\n",
    "* 904\n",
    "* 1204\n",
    "\n",
    "## Submit the results\n",
    "\n",
    "* Submit your results here: https://courses.datatalks.club/llm-zoomcamp-2024/homework/hw2\n",
    "* It's possible that your answers won't match exactly. If it's the case, select the closest one."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0510e72a-67bf-488b-9767-729342bf22e7",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d16db212-16e5-48fc-a9ef-64d4100c37c3",
   "metadata": {},
   "source": [
    "# Answers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6171afc-e900-4415-9628-4c2d8840637a",
   "metadata": {},
   "source": [
    "## Q1\n",
    "ollama version is 0.1.48"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6649fc1-f5db-4ecd-9109-928668fa572b",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "## Q2\n",
    "\n",
    "What I did:\n",
    "1. `docker start ollama`\n",
    "\n",
    "2. `docker exec -it ollama /bin/bash`\n",
    "\n",
    "3. `cd /root/.ollama/models/manifests/registry.ollama.ai/library/gemma`\n",
    "\n",
    "    Existing is `2b`.\n",
    "\n",
    "4. `cat 2b`\n",
    "   \n",
    "The content:\n",
    "\n",
    "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"digest\":\"sha256:887433b89a901c156f7e6944442f3c9e57f3c55d6ed52042cbb7303aea994290\",\"size\":483},\"layers\":[{\"mediaType\":\"application/vnd.ollama.image.model\",\"digest\":\"sha256:c1864a5eb19305c40519da12cc543519e48a0697ecd30e15d5ac228644957d12\",\"size\":1678447520},{\"mediaType\":\"application/vnd.ollama.image.license\",\"digest\":\"sha256:097a36493f718248845233af1d3fefe7a303f864fae13bc31a3a9704229378ca\",\"size\":8433},{\"mediaType\":\"application/vnd.ollama.image.template\",\"digest\":\"sha256:109037bec39c0becc8221222ae23557559bc594290945a2c4221ab4f303b8871\",\"size\":136},{\"mediaType\":\"application/vnd.ollama.image.params\",\"digest\":\"sha256:22a838ceb7fb22755a3b0ae9b4eadde629d19be1f651f73efb8c6b4e2cd0eea0\",\"size\":84}]}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9eb6c205-34b8-4ee9-908a-39dbbdeef044",
   "metadata": {},
   "source": [
    "## Q3\n",
    "`ollama run gemma:2b`\n",
    "\n",
    "`Answer:`\n",
    "\n",
    "Sure. Here's the answer to your question:\n",
    "\n",
    "10 * 10 = 100.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "455b8d92-9b74-4866-b76d-fe88aef419f8",
   "metadata": {},
   "source": [
    "## Q4: Downloading the weights\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ba4f1e-305e-4a60-9417-55ba2bfc1985",
   "metadata": {},
   "source": [
    "Size of the `ollama_files/models` folder: \n",
    "\n",
    "1,678,463,660 bytes (1.68 GB on disk) for 12 items\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae212826-9cc5-49b0-9e1b-22ea215bb356",
   "metadata": {},
   "source": [
    "## Q5: Adding the weights\n",
    "\n",
    "COPY models/blobs /root/.ollama/models/blobs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05f655cc-ac20-4153-b815-37df034ed352",
   "metadata": {},
   "source": [
    "## Q6: Serving it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "e1a52cc2-7c87-41d0-8043-ebb819b68ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv('/app/.env')\n",
    "api_key=os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "client = OpenAI(\n",
    "    base_url='http://localhost:11434/v1',\n",
    "    api_key='ollama',\n",
    ")\n",
    "\n",
    "# openai.api_base = 'http://localhost:11434/v1'\n",
    "# openai.api_key = os.getenv('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "51b25937-cd63-4c5f-b85f-d1fa20a90e4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_completion(prompt):\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"gemma:2b\",\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": prompt},\n",
    "        ],\n",
    "        temperature=0.0\n",
    "    )\n",
    "    response_text = response.choices[0].message.content\n",
    "    completion_tokens = response.usage.completion_tokens\n",
    "    return response_text, completion_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "b67c26dd-4864-4e45-9281-be9800c7d72c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response:\n",
      " Sure, here's the formula for energy:\n",
      "\n",
      "**E = m * c^2**\n",
      "\n",
      "Where:\n",
      "\n",
      "* **E** is the energy in joules (J)\n",
      "* **m** is the mass in kilograms (kg)\n",
      "* **c** is the speed of light in a vacuum squared (m/s^2)\n",
      "\n",
      "This formula tells us that energy is a measure of the amount of work that can be done or the amount of energy stored in an object.\n"
     ]
    }
   ],
   "source": [
    "prompt = \"What's the formula for energy?\"\n",
    "\n",
    "response_text, completion_tokens = get_completion(prompt)\n",
    "\n",
    "print(\"Response:\\n\", response_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "9761a001-63bb-403c-ba6d-bb049c62d1e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of completion tokens used:\n",
      " 101\n"
     ]
    }
   ],
   "source": [
    "### Counting the number of tokens\n",
    "print(\"Number of completion tokens used:\\n\", completion_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88d6b690-b507-4315-910b-fc51b0f73364",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}