tutorial_generative_ai_search.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "authorship_tag": "ABX9TyMbZnMInmklpi3UVhiPKjk+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Troyanovsky/Building-with-GenAI/blob/main/tutorial_generative_ai_search.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Build with GenAI: Generative AI Search with Local LLM\n",
        "\n",
        "- Local LLM (not using OpenAI's API). You can run the code on your own computer and keep everything private. Or you can use Google Colab's free T4 GPU (just hit Runtime - Change runtime type - T4 GPU; then you can run all cells.)\n",
        "- You can adapt the code easily to perform other tasks like searching on Arxiv, searching & summarizing local documents, etc.\n",
        "\n",
        "This Colab notebook is the accompanying code for my article at: https://medium.com/design-bootcamp/build-with-genai-generative-search-with-local-llm-342eb5a5037a\n",
        "\n",
        "\n",
        "This is part of the \"Build with GenAI\" series. Other tutorial projects can be found at: https://github.com/Troyanovsky/Building-with-GenAI/tree/main"
      ],
      "metadata": {
        "id": "xcKlbG02Pix_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Install llama-cpp-python\n",
        "!CMAKE_ARGS=\"-DLLAMA_CUBLAS=on\" FORCE_CMAKE=1 pip install llama-cpp-python --no-cache-dir\n",
        "\n",
        "%cd /content\n",
        "!apt-get update -qq && apt-get install -y -qq aria2\n",
        "\n",
        "# Download a local large language model, I'm using OpernHermes-2.5-Mistral-7B-16K-GGUF which has a longer context size and has pretty good quality at its size\n",
        "# If you want to use other local models that can easily run on consumer hardware, check out this repo: https://github.com/Troyanovsky/Local-LLM-Comparison-Colab-UI/\n",
        "!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/TheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF/resolve/main/openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf?download=true -d /content/model/ -o openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf"
      ],
      "metadata": {
        "id": "DIJ_lk4bSlt7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bb776af4-1574-49d8-fce9-6b9c2c556e57"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting llama-cpp-python\n",
            "  Downloading llama_cpp_python-0.2.63.tar.gz (37.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m37.5/37.5 MB\u001b[0m \u001b[31m79.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: typing-extensions>=4.5.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (4.11.0)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (1.25.2)\n",
            "Collecting diskcache>=5.6.1 (from llama-cpp-python)\n",
            "  Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.5/45.5 kB\u001b[0m \u001b[31m195.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: jinja2>=2.11.3 in /usr/local/lib/python3.10/dist-packages (from llama-cpp-python) (3.1.3)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2>=2.11.3->llama-cpp-python) (2.1.5)\n",
            "Building wheels for collected packages: llama-cpp-python\n",
            "  Building wheel for llama-cpp-python (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.2.63-cp310-cp310-linux_x86_64.whl size=39168030 sha256=1b3a605d471975601119bf1ce3a515a808104258e5601568cdbcf4b804f56c5f\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-d06g2glq/wheels/5c/32/58/9b1aa714aea4ce0f0e0ee441f40d35c869aa6d40296437c796\n",
            "Successfully built llama-cpp-python\n",
            "Installing collected packages: diskcache, llama-cpp-python\n",
            "Successfully installed diskcache-5.6.3 llama-cpp-python-0.2.63\n",
            "/content\n",
            "Selecting previously unselected package libc-ares2:amd64.\n",
            "(Reading database ... 131015 files and directories currently installed.)\n",
            "Preparing to unpack .../libc-ares2_1.18.1-1ubuntu0.22.04.3_amd64.deb ...\n",
            "Unpacking libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Selecting previously unselected package libaria2-0:amd64.\n",
            "Preparing to unpack .../libaria2-0_1.36.0-1_amd64.deb ...\n",
            "Unpacking libaria2-0:amd64 (1.36.0-1) ...\n",
            "Selecting previously unselected package aria2.\n",
            "Preparing to unpack .../aria2_1.36.0-1_amd64.deb ...\n",
            "Unpacking aria2 (1.36.0-1) ...\n",
            "Setting up libc-ares2:amd64 (1.18.1-1ubuntu0.22.04.3) ...\n",
            "Setting up libaria2-0:amd64 (1.36.0-1) ...\n",
            "Setting up aria2 (1.36.0-1) ...\n",
            "Processing triggers for man-db (2.10.2-1) ...\n",
            "Processing triggers for libc-bin (2.35-0ubuntu3.4) ...\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_5.so.3 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbmalloc_proxy.so.2 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbb.so.12 is not a symbolic link\n",
            "\n",
            "/sbin/ldconfig.real: /usr/local/lib/libtbbbind_2_0.so.3 is not a symbolic link\n",
            "\n",
            " *** Download Progress Summary as of Mon Apr 22 10:45:37 2024 *** \n",
            "=\n",
            "[#fa3274 1.0GiB/4.0GiB(27%) CN:16 DL:75MiB ETA:40s]\n",
            "FILE: /content/model//openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf\n",
            "-\n",
            "\n",
            "\u001b[0m\n",
            "Download Results:\n",
            "gid   |stat|avg speed  |path/URI\n",
            "======+====+===========+=======================================================\n",
            "fa3274|\u001b[1;32mOK\u001b[0m  |    79MiB/s|/content/model//openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf\n",
            "\n",
            "Status Legend:\n",
            "(OK):download completed.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rTKrS7XSOcR7"
      },
      "outputs": [],
      "source": [
        "import requests\n",
        "import subprocess\n",
        "import json\n",
        "import time\n",
        "\n",
        "# Function for calling search API\n",
        "def get_search_results(search_term, max_retries=2, retry_delay=2):\n",
        "    url = \"https://google.serper.dev/search\"\n",
        "    payload = json.dumps({\"q\": search_term})\n",
        "    headers = {\n",
        "        'X-API-KEY': '<your_api_key>', # Replace with your own API Key\n",
        "        'Content-Type': 'application/json'\n",
        "    }\n",
        "\n",
        "    retries = 0\n",
        "    while retries < max_retries:\n",
        "        try:\n",
        "            response = requests.request(\"POST\", url, headers=headers, data=payload)\n",
        "            response.raise_for_status()  # Raise an exception for non-2xx status codes\n",
        "            data = response.json()\n",
        "            organic_results = data.get(\"organic\", [])\n",
        "\n",
        "            search_results = []\n",
        "            search_results_str = \"\"\n",
        "            index = 0\n",
        "            for result in organic_results:\n",
        "                title = result.get(\"title\", \"\")\n",
        "                link = result.get(\"link\", \"\")\n",
        "                snippet = result.get(\"snippet\", \"\")\n",
        "                search_results.append({\"title\": title, \"link\": link, \"snippet\": snippet})\n",
        "                formatted_result = f\"index: {index}\\ntitle: {title}\\nlink: {link}\\nsnippet: {snippet}\\n\\n\"\n",
        "                search_results_str += formatted_result\n",
        "                index += 1\n",
        "            return search_results, search_results_str\n",
        "        except requests.exceptions.RequestException as e:\n",
        "            retries += 1\n",
        "            print(f\"Error: {e}. Retrying in {retry_delay} seconds... (Attempt {retries}/{max_retries})\")\n",
        "            time.sleep(retry_delay)\n",
        "\n",
        "    raise Exception(\"Maximum retries exceeded. Failed to retrieve search results.\")\n",
        "\n",
        "\n",
        "def fetch_url_content(url):\n",
        "    # Prepend \"https://r.jina.ai/\" to the input URL\n",
        "    # This converts the URL into LLM-friendly format. Check out their GitHub: https://github.com/jina-ai/reader\n",
        "    prefixed_url = f\"https://r.jina.ai/{url}\"\n",
        "\n",
        "\n",
        "    try:\n",
        "        curl_cmd = [\n",
        "            \"curl\",\n",
        "            \"-H\",\n",
        "            \"Accept: text/event-stream\",\n",
        "            prefixed_url,\n",
        "        ]\n",
        "        curl_process = subprocess.Popen(curl_cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n",
        "        stdout, stderr = curl_process.communicate()\n",
        "\n",
        "        if curl_process.returncode == 0:\n",
        "            content = stdout.decode(\"utf-8\")\n",
        "\n",
        "            content_lines = [line for line in content.split(\"\\n\") if line.startswith(\"data: \")]\n",
        "            if content_lines:\n",
        "                content_data = \"\\n\".join(line[6:] for line in content_lines)\n",
        "                try:\n",
        "                    content_value = json.loads(content_data)[\"content\"]\n",
        "                    return content_value\n",
        "                except (ValueError, KeyError):\n",
        "                    pass\n",
        "\n",
        "            return \"\"\n",
        "        else:\n",
        "            error_message = stderr.decode(\"utf-8\")\n",
        "            raise Exception(f\"cURL request failed: {error_message}\")\n",
        "\n",
        "    except Exception as e:\n",
        "        raise Exception(f\"An error occurred: {e}\")"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Setting up a local LLM for summarization or chat\n",
        "from llama_cpp import Llama\n",
        "\n",
        "def load_llama():\n",
        "    llm = Llama(\n",
        "            model_path=\"/content/model/openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf\", # If you're using another model, change the name\n",
        "            chat_format=\"chatml\", # Use the chat_format that matches the model\n",
        "            n_gpu_layers=-1, # Use -1 for all layers on GPU\n",
        "            n_ctx=12288 # Set context size\n",
        "    )\n",
        "    return llm\n",
        "\n",
        "def call_llama(input: str, llm) -> str:\n",
        "    llm = llm\n",
        "    output = llm.create_chat_completion(\n",
        "        messages=[\n",
        "            {\n",
        "                \"role\": \"system\",\n",
        "                \"content\": \"You're a helpful assistant.\",\n",
        "            }, # Feel free to modify the prompt to suit your own formatting needs\n",
        "            {\"role\": \"user\", \"content\": input},\n",
        "        ],\n",
        "        temperature=0.7,\n",
        "    )\n",
        "    output_text = output['choices'][0]['message']['content']\n",
        "    return output_text"
      ],
      "metadata": {
        "id": "QBo4bIOnTBsC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def pick_url(query, search_results_str, search_results, llm):\n",
        "    llm = llm\n",
        "    prompt = f\"Given the following question, which of the following URLs is most likely to contain the answer for it? Reply ONLY the index number. Question: ```{query}``` List: ```{search_results_str}```\"\n",
        "    index = call_llama(prompt, llm)\n",
        "\n",
        "    max_retries = 2\n",
        "    retries = 0\n",
        "    while retries < max_retries:\n",
        "        try:\n",
        "            index = int(index.strip())\n",
        "            break\n",
        "        except ValueError:\n",
        "            retries += 1\n",
        "            index = call_llama(prompt, llm)\n",
        "\n",
        "    if retries == max_retries:\n",
        "        raise Exception(\"Failed to convert index to a valid integer after multiple retries.\")\n",
        "\n",
        "    try:\n",
        "        return index\n",
        "    except IndexError:\n",
        "        raise Exception(f\"Invalid index {index} for the search results list.\")"
      ],
      "metadata": {
        "id": "EbAvrHrNYDZm"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def search_with_ai(user_input):\n",
        "    llm = None\n",
        "\n",
        "    llm = load_llama()\n",
        "\n",
        "    search_term_prompt = f\"Based on the following question, plesae come up with a search term to use in the search engine. Reply the search term only. Quesiton: ```{user_input}```\"\n",
        "    search_term = call_llama(search_term_prompt, llm)\n",
        "    print(f\"Searching: {search_term}\")\n",
        "\n",
        "    # Seach with search API\n",
        "    search_results, search_results_str = get_search_results(search_term)\n",
        "\n",
        "    # Pick the most relevant URL\n",
        "    try:\n",
        "        top_url_index = pick_url(user_input, search_results_str, search_results, llm)\n",
        "    except Exception as e:\n",
        "        print(f\"Error picking URL: {e}\")\n",
        "        return\n",
        "\n",
        "    # Fetch the content from the top URL\n",
        "    try:\n",
        "        top_url = search_results[top_url_index][\"link\"]\n",
        "        top_snippet = search_results[top_url_index][\"snippet\"]\n",
        "        print(f\"Crawling: {top_url}\")\n",
        "        content = fetch_url_content(top_url)\n",
        "    except Exception as e:\n",
        "        print(f\"Error fetching URL content: {e}\")\n",
        "        del llm\n",
        "        return\n",
        "\n",
        "    # Truncate the content if it's longer than 36864 characters. I'm using a very lazy estimate here. You can count actual tokens instead.\n",
        "    if len(content) > 36864:\n",
        "        content = content[:36864]\n",
        "\n",
        "    # Call LLM with the content and get the answer\n",
        "    answer_prompt = f\"Answer the question from the given content. Question: ```{user_input}```\\n\\nContent:```From URL: {top_url} Snippet: {top_snippet}\\n{content}```\"\n",
        "    try:\n",
        "        answer = call_llama(answer_prompt, llm)\n",
        "        return answer\n",
        "    except Exception as e:\n",
        "        print(f\"Error calling LLM: {e}\")\n",
        "        return"
      ],
      "metadata": {
        "id": "i_75BFf3XVL2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "question = input(\"What is your question? \\n\")\n",
        "answer = search_with_ai(question)\n",
        "print(answer)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "794h-hEjXpLs",
        "outputId": "002aedb0-c50c-486f-f49c-61b74a4b5331"
      },
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "What is your question? What is Llama-3? When is it released?\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "llama_model_loader: loaded meta data with 23 key-value pairs and 291 tensors from /content/model/openhermes-2.5-mistral-7b-16k.Q4_K_M.gguf (version GGUF V3 (latest))\n",
            "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
            "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
            "llama_model_loader: - kv   1:                               general.name str              = nurtureai_openhermes-2.5-mistral-7b-16k\n",
            "llama_model_loader: - kv   2:                       llama.context_length u32              = 32768\n",
            "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096\n",
            "llama_model_loader: - kv   4:                          llama.block_count u32              = 32\n",
            "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336\n",
            "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
            "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32\n",
            "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8\n",
            "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
            "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 100000.000000\n",
            "llama_model_loader: - kv  11:                          general.file_type u32              = 15\n",
            "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
            "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32002]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
            "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32002]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
            "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32002]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
            "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
            "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 32000\n",
            "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
            "llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 0\n",
            "llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true\n",
            "llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false\n",
            "llama_model_loader: - kv  22:               general.quantization_version u32              = 2\n",
            "llama_model_loader: - type  f32:   65 tensors\n",
            "llama_model_loader: - type q4_K:  193 tensors\n",
            "llama_model_loader: - type q6_K:   33 tensors\n",
            "llm_load_vocab: special tokens definition check successful ( 261/32002 ).\n",
            "llm_load_print_meta: format           = GGUF V3 (latest)\n",
            "llm_load_print_meta: arch             = llama\n",
            "llm_load_print_meta: vocab type       = SPM\n",
            "llm_load_print_meta: n_vocab          = 32002\n",
            "llm_load_print_meta: n_merges         = 0\n",
            "llm_load_print_meta: n_ctx_train      = 32768\n",
            "llm_load_print_meta: n_embd           = 4096\n",
            "llm_load_print_meta: n_head           = 32\n",
            "llm_load_print_meta: n_head_kv        = 8\n",
            "llm_load_print_meta: n_layer          = 32\n",
            "llm_load_print_meta: n_rot            = 128\n",
            "llm_load_print_meta: n_embd_head_k    = 128\n",
            "llm_load_print_meta: n_embd_head_v    = 128\n",
            "llm_load_print_meta: n_gqa            = 4\n",
            "llm_load_print_meta: n_embd_k_gqa     = 1024\n",
            "llm_load_print_meta: n_embd_v_gqa     = 1024\n",
            "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
            "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
            "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
            "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
            "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
            "llm_load_print_meta: n_ff             = 14336\n",
            "llm_load_print_meta: n_expert         = 0\n",
            "llm_load_print_meta: n_expert_used    = 0\n",
            "llm_load_print_meta: causal attn      = 1\n",
            "llm_load_print_meta: pooling type     = 0\n",
            "llm_load_print_meta: rope type        = 0\n",
            "llm_load_print_meta: rope scaling     = linear\n",
            "llm_load_print_meta: freq_base_train  = 100000.0\n",
            "llm_load_print_meta: freq_scale_train = 1\n",
            "llm_load_print_meta: n_yarn_orig_ctx  = 32768\n",
            "llm_load_print_meta: rope_finetuned   = unknown\n",
            "llm_load_print_meta: ssm_d_conv       = 0\n",
            "llm_load_print_meta: ssm_d_inner      = 0\n",
            "llm_load_print_meta: ssm_d_state      = 0\n",
            "llm_load_print_meta: ssm_dt_rank      = 0\n",
            "llm_load_print_meta: model type       = 7B\n",
            "llm_load_print_meta: model ftype      = Q4_K - Medium\n",
            "llm_load_print_meta: model params     = 7.24 B\n",
            "llm_load_print_meta: model size       = 4.07 GiB (4.83 BPW) \n",
            "llm_load_print_meta: general.name     = nurtureai_openhermes-2.5-mistral-7b-16k\n",
            "llm_load_print_meta: BOS token        = 1 '<s>'\n",
            "llm_load_print_meta: EOS token        = 32000 '<|im_end|>'\n",
            "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
            "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
            "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
            "llm_load_tensors: ggml ctx size =    0.30 MiB\n",
            "llm_load_tensors: offloading 32 repeating layers to GPU\n",
            "llm_load_tensors: offloading non-repeating layers to GPU\n",
            "llm_load_tensors: offloaded 33/33 layers to GPU\n",
            "llm_load_tensors:        CPU buffer size =    70.32 MiB\n",
            "llm_load_tensors:      CUDA0 buffer size =  4095.06 MiB\n",
            "................................................................................................\n",
            "llama_new_context_with_model: n_ctx      = 12288\n",
            "llama_new_context_with_model: n_batch    = 512\n",
            "llama_new_context_with_model: n_ubatch   = 512\n",
            "llama_new_context_with_model: freq_base  = 100000.0\n",
            "llama_new_context_with_model: freq_scale = 1\n",
            "llama_kv_cache_init:      CUDA0 KV buffer size =  1536.00 MiB\n",
            "llama_new_context_with_model: KV self size  = 1536.00 MiB, K (f16):  768.00 MiB, V (f16):  768.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host  output buffer size =     0.12 MiB\n",
            "llama_new_context_with_model:      CUDA0 compute buffer size =   824.00 MiB\n",
            "llama_new_context_with_model:  CUDA_Host compute buffer size =    32.01 MiB\n",
            "llama_new_context_with_model: graph nodes  = 1030\n",
            "llama_new_context_with_model: graph splits = 2\n",
            "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
            "Model metadata: {'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.unknown_token_id': '0', 'tokenizer.ggml.eos_token_id': '32000', 'general.architecture': 'llama', 'llama.rope.freq_base': '100000.000000', 'llama.context_length': '32768', 'general.name': 'nurtureai_openhermes-2.5-mistral-7b-16k', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '32', 'llama.block_count': '32', 'llama.attention.head_count_kv': '8', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '15'}\n",
            "\n",
            "llama_print_timings:        load time =     245.22 ms\n",
            "llama_print_timings:      sample time =       5.97 ms /    10 runs   (    0.60 ms per token,  1675.32 tokens per second)\n",
            "llama_print_timings: prompt eval time =     244.78 ms /    72 tokens (    3.40 ms per token,   294.15 tokens per second)\n",
            "llama_print_timings:        eval time =     232.74 ms /     9 runs   (   25.86 ms per token,    38.67 tokens per second)\n",
            "llama_print_timings:       total time =     512.86 ms /    81 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\"Llama-3 release date\"\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =     245.22 ms\n",
            "llama_print_timings:      sample time =       1.30 ms /     2 runs   (    0.65 ms per token,  1539.65 tokens per second)\n",
            "llama_print_timings: prompt eval time =    1299.16 ms /  1079 tokens (    1.20 ms per token,   830.54 tokens per second)\n",
            "llama_print_timings:        eval time =      26.44 ms /     1 runs   (   26.44 ms per token,    37.82 tokens per second)\n",
            "llama_print_timings:       total time =    1349.01 ms /  1080 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "https://www.geeksforgeeks.org/llama-3-metas-new-ai-model/\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Llama.generate: prefix-match hit\n",
            "\n",
            "llama_print_timings:        load time =     245.22 ms\n",
            "llama_print_timings:      sample time =     102.07 ms /   157 runs   (    0.65 ms per token,  1538.13 tokens per second)\n",
            "llama_print_timings: prompt eval time =    6396.92 ms /  4822 tokens (    1.33 ms per token,   753.80 tokens per second)\n",
            "llama_print_timings:        eval time =    5685.59 ms /   156 runs   (   36.45 ms per token,    27.44 tokens per second)\n",
            "llama_print_timings:       total time =   12852.58 ms /  4978 tokens\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Llama-3 is Meta's latest and most powerful large language model (LLM). It was released on April 18, 2024. It uses a powerful tokenizer with a vocabulary of 128,000 tokens and is trained on 15 trillion tokens, making it 7 times larger than its predecessor Llama-2. Llama-3 excels at understanding language, enhancing the performance of Meta's platforms like Facebook, Instagram, WhatsApp, and Messenger. It offers features such as improved creativity, increased productivity, accessibility improvements, and search integration within the apps. The full open-source model is expected to be released in July 2024.\n"
          ]
        }
      ]
    }
  ]
}