examples/python/llama.cpp/llama.cpp_in_Spark_NLP_AutoGGUFModel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![JohnSnowLabs](https://sparknlp.org/assets/images/logo.png)\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/JohnSnowLabs/spark-nlp/blob/master/examples/python/llama.cpp/llama.cpp_in_Spark_NLP_AutoGGUFModel.ipynb)\n",
    "\n",
    "# Import llama.cpp 🦙 models into Spark NLP 🚀\n",
    "\n",
    "Let's keep in mind a few things before we start 😊\n",
    "\n",
    "- llama.cpp support was introduced in `Spark NLP 5.5.0`, enabling quantized LLM inference on a wide range of devices. Please make sure you have upgraded to the latest Spark NLP release.\n",
    "- You need to use your own `.gguf` model files, which also include the models from the [Hugging Face Models](https://huggingface.co/models?library=gguf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download a GGUF Model\n",
    "\n",
    "Lets download a GGUF model to test it out. For this, we will use [bartowski/Phi-3.5-mini-instruct-GGUF](https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF). It is a 3.8B parameter model which also is available in 4-bit quantization. \n",
    "\n",
    "We can download the model by selecting the q4 GGUF file from the \"Files and versions\" tab.\n",
    "\n",
    "Once downloaded, we can directly import this model into Spark NLP!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-07-20 11:11:30--  https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true\n",
      "Resolving huggingface.co (huggingface.co)... 2600:9000:275f:7600:17:b174:6d00:93a1, 2600:9000:275f:3800:17:b174:6d00:93a1, 2600:9000:275f:6e00:17:b174:6d00:93a1, ...\n",
      "Connecting to huggingface.co (huggingface.co)|2600:9000:275f:7600:17:b174:6d00:93a1|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3.5-mini-instruct-Q4_K_M.gguf%3B+filename%3D%22P Phi-3.5-mini-instruct-Q4_K_M.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9 [following]\n",
      "--2024-07-20 11:11:30--  https://cdn-lfs-us-1.huggingface.co/repos/41/c8/41c860f65b01de5dc4c68b00d84cead799d3e7c48e38ee749f4c6057776e2e9e/8a83c7fb9049a9b2e92266fa7ad04933bb53aa1e85136b7b30f1b8000ff2edef?response-content-disposition=attachment%3B+filename*%3DUTF-8%27%27Phi-3.5-mini-instruct-Q4_K_M.gguf%3B+filename%3D%22P Phi-3.5-mini-instruct-Q4_K_M.gguf%22%3B&Expires=1721725890&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTcyMTcyNTg5MH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmh1Z2dpbmdmYWNlLmNvL3JlcG9zLzQxL2M4LzQxYzg2MGY2NWIwMWRlNWRjNGM2OGIwMGQ4NGNlYWQ3OTlkM2U3YzQ4ZTM4ZWU3NDlmNGM2MDU3Nzc2ZTJlOWUvOGE4M2M3ZmI5MDQ5YTliMmU5MjI2NmZhN2FkMDQ5MzNiYjUzYWExZTg1MTM2YjdiMzBmMWI4MDAwZmYyZWRlZj9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=joXQf4QRpEhtFeQ3r3gJ0zyJ3bXReb9OxM%7EZit3GJ3355ycKQzemJ%7E6eD-J7%7EkphnsPpRpUDhQkCr2-Oidqo7dgltmFsWAX4SmQLn65R1yjO%7EsMvi%7E4vOUpaRPYlSMCyWWJpiZZjQYVH4Uk0o-G62ALFXKGaDfr627kvahP-fJYwNNP1riTrH8hbbah28ZKRAQjUGI1aNqerG0jojudnGOagawISAnudkAOFZfxnN7Qw3CoMySZLj9Euu02RBv2A5Yy0uSjG7b8rilx-tU5HDR3ECohdQQ8yPXjYFU-LZi-zcG1wwBDF-S01qb%7EgPWsTorenxfRM2cG6J%7EvSziGCzA__&Key-Pair-Id=K24J24Z295AEI9\n",
      "Resolving cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)... 2600:9000:275f:7a00:17:9a40:4dc0:93a1, 2600:9000:275f:fc00:17:9a40:4dc0:93a1, 2600:9000:275f:4800:17:9a40:4dc0:93a1, ...\n",
      "Connecting to cdn-lfs-us-1.huggingface.co (cdn-lfs-us-1.huggingface.co)|2600:9000:275f:7a00:17:9a40:4dc0:93a1|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2393231072 (2.2G) [binary/octet-stream]\n",
      "Saving to: ‘Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true’\n",
      "\n",
      "Phi-3-mini-4k-instr 100%[===================>]   2.23G  22.5MB/s    in 96s     \n",
      "\n",
      "2024-07-20 11:13:06 (23.7 MB/s) - ‘Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true’ saved [2393231072/2393231072]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "EXPORT_PATH = \"Phi-3.5-mini-instruct-Q4_K_M.gguf\"\n",
    "! wget \"https://huggingface.co/bartowski/Phi-3.5-mini-instruct-GGUF/resolve/main/Phi-3.5-mini-instruct-Q4_K_M.gguf?download=true\" -O  {EXPORT_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import and Save AutGGUF models in Spark NLP\n",
    "\n",
    "- Let's install and setup Spark NLP (if running it Google Colab)\n",
    "- This part is pretty easy via our simple script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only execute this if you are on Google Colab\n",
    "! wget -q http://setup.johnsnowlabs.com/colab.sh -O - | bash"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start Spark with Spark NLP included via our simple `start()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "24/07/21 10:51:01 WARN Utils: Your hostname, pop-os resolves to a loopback address: 127.0.1.1; using 192.168.0.34 instead (on interface enp3s0)\n",
      "24/07/21 10:51:01 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ":: loading settings :: url = jar:file:/home/ducha/mambaforge/envs/sparknlp_dev/lib/python3.10/site-packages/pyspark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Ivy Default Cache set to: /home/ducha/.ivy2/cache\n",
      "The jars for the packages stored in: /home/ducha/.ivy2/jars\n",
      "com.johnsnowlabs.nlp#spark-nlp_2.12 added as a dependency\n",
      ":: resolving dependencies :: org.apache.spark#spark-submit-parent-994cb793-bb56-4b46-ad2f-b20d68529970;1.0\n",
      "\tconfs: [default]\n",
      "\tfound com.johnsnowlabs.nlp#spark-nlp_2.12;5.3.3 in central\n",
      "\tfound com.typesafe#config;1.4.2 in central\n",
      "\tfound org.rocksdb#rocksdbjni;6.29.5 in central\n",
      "\tfound com.amazonaws#aws-java-sdk-s3;1.12.500 in central\n",
      "\tfound com.amazonaws#aws-java-sdk-kms;1.12.500 in central\n",
      "\tfound com.amazonaws#aws-java-sdk-core;1.12.500 in central\n",
      "\tfound commons-logging#commons-logging;1.1.3 in central\n",
      "\tfound commons-codec#commons-codec;1.15 in central\n",
      "\tfound org.apache.httpcomponents#httpclient;4.5.13 in central\n",
      "\tfound org.apache.httpcomponents#httpcore;4.4.13 in central\n",
      "\tfound software.amazon.ion#ion-java;1.0.2 in central\n",
      "\tfound joda-time#joda-time;2.8.1 in central\n",
      "\tfound com.amazonaws#jmespath-java;1.12.500 in central\n",
      "\tfound com.github.universal-automata#liblevenshtein;3.0.0 in central\n",
      "\tfound com.google.protobuf#protobuf-java-util;3.0.0-beta-3 in central\n",
      "\tfound com.google.protobuf#protobuf-java;3.0.0-beta-3 in central\n",
      "\tfound com.google.code.gson#gson;2.3 in central\n",
      "\tfound it.unimi.dsi#fastutil;7.0.12 in central\n",
      "\tfound org.projectlombok#lombok;1.16.8 in central\n",
      "\tfound com.google.cloud#google-cloud-storage;2.20.1 in central\n",
      "\tfound com.google.guava#guava;31.1-jre in central\n",
      "\tfound com.google.guava#failureaccess;1.0.1 in central\n",
      "\tfound com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava in central\n",
      "\tfound com.google.errorprone#error_prone_annotations;2.18.0 in central\n",
      "\tfound com.google.j2objc#j2objc-annotations;1.3 in central\n",
      "\tfound com.google.http-client#google-http-client;1.43.0 in central\n",
      "\tfound io.opencensus#opencensus-contrib-http-util;0.31.1 in central\n",
      "\tfound com.google.http-client#google-http-client-jackson2;1.43.0 in central\n",
      "\tfound com.google.http-client#google-http-client-gson;1.43.0 in central\n",
      "\tfound com.google.api-client#google-api-client;2.2.0 in central\n",
      "\tfound com.google.oauth-client#google-oauth-client;1.34.1 in central\n",
      "\tfound com.google.http-client#google-http-client-apache-v2;1.43.0 in central\n",
      "\tfound com.google.apis#google-api-services-storage;v1-rev20220705-2.0.0 in central\n",
      "\tfound com.google.code.gson#gson;2.10.1 in central\n",
      "\tfound com.google.cloud#google-cloud-core;2.12.0 in central\n",
      "\tfound io.grpc#grpc-context;1.53.0 in central\n",
      "\tfound com.google.auto.value#auto-value-annotations;1.10.1 in central\n",
      "\tfound com.google.auto.value#auto-value;1.10.1 in central\n",
      "\tfound javax.annotation#javax.annotation-api;1.3.2 in central\n",
      "\tfound com.google.cloud#google-cloud-core-http;2.12.0 in central\n",
      "\tfound com.google.http-client#google-http-client-appengine;1.43.0 in central\n",
      "\tfound com.google.api#gax-httpjson;0.108.2 in central\n",
      "\tfound com.google.cloud#google-cloud-core-grpc;2.12.0 in central\n",
      "\tfound io.grpc#grpc-alts;1.53.0 in central\n",
      "\tfound io.grpc#grpc-grpclb;1.53.0 in central\n",
      "\tfound org.conscrypt#conscrypt-openjdk-uber;2.5.2 in central\n",
      "\tfound io.grpc#grpc-auth;1.53.0 in central\n",
      "\tfound io.grpc#grpc-protobuf;1.53.0 in central\n",
      "\tfound io.grpc#grpc-protobuf-lite;1.53.0 in central\n",
      "\tfound io.grpc#grpc-core;1.53.0 in central\n",
      "\tfound com.google.api#gax;2.23.2 in central\n",
      "\tfound com.google.api#gax-grpc;2.23.2 in central\n",
      "\tfound com.google.auth#google-auth-library-credentials;1.16.0 in central\n",
      "\tfound com.google.auth#google-auth-library-oauth2-http;1.16.0 in central\n",
      "\tfound com.google.api#api-common;2.6.2 in central\n",
      "\tfound io.opencensus#opencensus-api;0.31.1 in central\n",
      "\tfound com.google.api.grpc#proto-google-iam-v1;1.9.2 in central\n",
      "\tfound com.google.protobuf#protobuf-java;3.21.12 in central\n",
      "\tfound com.google.protobuf#protobuf-java-util;3.21.12 in central\n",
      "\tfound com.google.api.grpc#proto-google-common-protos;2.14.2 in central\n",
      "\tfound org.threeten#threetenbp;1.6.5 in central\n",
      "\tfound com.google.api.grpc#proto-google-cloud-storage-v2;2.20.1-alpha in central\n",
      "\tfound com.google.api.grpc#grpc-google-cloud-storage-v2;2.20.1-alpha in central\n",
      "\tfound com.google.api.grpc#gapic-google-cloud-storage-v2;2.20.1-alpha in central\n",
      "\tfound com.google.code.findbugs#jsr305;3.0.2 in central\n",
      "\tfound io.grpc#grpc-api;1.53.0 in central\n",
      "\tfound io.grpc#grpc-stub;1.53.0 in central\n",
      "\tfound org.checkerframework#checker-qual;3.31.0 in central\n",
      "\tfound io.perfmark#perfmark-api;0.26.0 in central\n",
      "\tfound com.google.android#annotations;4.1.1.4 in central\n",
      "\tfound org.codehaus.mojo#animal-sniffer-annotations;1.22 in central\n",
      "\tfound io.opencensus#opencensus-proto;0.2.0 in central\n",
      "\tfound io.grpc#grpc-services;1.53.0 in central\n",
      "\tfound com.google.re2j#re2j;1.6 in central\n",
      "\tfound io.grpc#grpc-netty-shaded;1.53.0 in central\n",
      "\tfound io.grpc#grpc-googleapis;1.53.0 in central\n",
      "\tfound io.grpc#grpc-xds;1.53.0 in central\n",
      "\tfound com.navigamez#greex;1.0 in central\n",
      "\tfound dk.brics.automaton#automaton;1.11-8 in central\n",
      "\tfound com.johnsnowlabs.nlp#tensorflow-cpu_2.12;0.4.4 in central\n",
      "\tfound com.microsoft.onnxruntime#onnxruntime;1.17.0 in central\n",
      ":: resolution report :: resolve 843ms :: artifacts dl 40ms\n",
      "\t:: modules in use:\n",
      "\tcom.amazonaws#aws-java-sdk-core;1.12.500 from central in [default]\n",
      "\tcom.amazonaws#aws-java-sdk-kms;1.12.500 from central in [default]\n",
      "\tcom.amazonaws#aws-java-sdk-s3;1.12.500 from central in [default]\n",
      "\tcom.amazonaws#jmespath-java;1.12.500 from central in [default]\n",
      "\tcom.github.universal-automata#liblevenshtein;3.0.0 from central in [default]\n",
      "\tcom.google.android#annotations;4.1.1.4 from central in [default]\n",
      "\tcom.google.api#api-common;2.6.2 from central in [default]\n",
      "\tcom.google.api#gax;2.23.2 from central in [default]\n",
      "\tcom.google.api#gax-grpc;2.23.2 from central in [default]\n",
      "\tcom.google.api#gax-httpjson;0.108.2 from central in [default]\n",
      "\tcom.google.api-client#google-api-client;2.2.0 from central in [default]\n",
      "\tcom.google.api.grpc#gapic-google-cloud-storage-v2;2.20.1-alpha from central in [default]\n",
      "\tcom.google.api.grpc#grpc-google-cloud-storage-v2;2.20.1-alpha from central in [default]\n",
      "\tcom.google.api.grpc#proto-google-cloud-storage-v2;2.20.1-alpha from central in [default]\n",
      "\tcom.google.api.grpc#proto-google-common-protos;2.14.2 from central in [default]\n",
      "\tcom.google.api.grpc#proto-google-iam-v1;1.9.2 from central in [default]\n",
      "\tcom.google.apis#google-api-services-storage;v1-rev20220705-2.0.0 from central in [default]\n",
      "\tcom.google.auth#google-auth-library-credentials;1.16.0 from central in [default]\n",
      "\tcom.google.auth#google-auth-library-oauth2-http;1.16.0 from central in [default]\n",
      "\tcom.google.auto.value#auto-value;1.10.1 from central in [default]\n",
      "\tcom.google.auto.value#auto-value-annotations;1.10.1 from central in [default]\n",
      "\tcom.google.cloud#google-cloud-core;2.12.0 from central in [default]\n",
      "\tcom.google.cloud#google-cloud-core-grpc;2.12.0 from central in [default]\n",
      "\tcom.google.cloud#google-cloud-core-http;2.12.0 from central in [default]\n",
      "\tcom.google.cloud#google-cloud-storage;2.20.1 from central in [default]\n",
      "\tcom.google.code.findbugs#jsr305;3.0.2 from central in [default]\n",
      "\tcom.google.code.gson#gson;2.10.1 from central in [default]\n",
      "\tcom.google.errorprone#error_prone_annotations;2.18.0 from central in [default]\n",
      "\tcom.google.guava#failureaccess;1.0.1 from central in [default]\n",
      "\tcom.google.guava#guava;31.1-jre from central in [default]\n",
      "\tcom.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava from central in [default]\n",
      "\tcom.google.http-client#google-http-client;1.43.0 from central in [default]\n",
      "\tcom.google.http-client#google-http-client-apache-v2;1.43.0 from central in [default]\n",
      "\tcom.google.http-client#google-http-client-appengine;1.43.0 from central in [default]\n",
      "\tcom.google.http-client#google-http-client-gson;1.43.0 from central in [default]\n",
      "\tcom.google.http-client#google-http-client-jackson2;1.43.0 from central in [default]\n",
      "\tcom.google.j2objc#j2objc-annotations;1.3 from central in [default]\n",
      "\tcom.google.oauth-client#google-oauth-client;1.34.1 from central in [default]\n",
      "\tcom.google.protobuf#protobuf-java;3.21.12 from central in [default]\n",
      "\tcom.google.protobuf#protobuf-java-util;3.21.12 from central in [default]\n",
      "\tcom.google.re2j#re2j;1.6 from central in [default]\n",
      "\tcom.johnsnowlabs.nlp#spark-nlp_2.12;5.3.3 from central in [default]\n",
      "\tcom.johnsnowlabs.nlp#tensorflow-cpu_2.12;0.4.4 from central in [default]\n",
      "\tcom.microsoft.onnxruntime#onnxruntime;1.17.0 from central in [default]\n",
      "\tcom.navigamez#greex;1.0 from central in [default]\n",
      "\tcom.typesafe#config;1.4.2 from central in [default]\n",
      "\tcommons-codec#commons-codec;1.15 from central in [default]\n",
      "\tcommons-logging#commons-logging;1.1.3 from central in [default]\n",
      "\tdk.brics.automaton#automaton;1.11-8 from central in [default]\n",
      "\tio.grpc#grpc-alts;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-api;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-auth;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-context;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-core;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-googleapis;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-grpclb;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-netty-shaded;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-protobuf;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-protobuf-lite;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-services;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-stub;1.53.0 from central in [default]\n",
      "\tio.grpc#grpc-xds;1.53.0 from central in [default]\n",
      "\tio.opencensus#opencensus-api;0.31.1 from central in [default]\n",
      "\tio.opencensus#opencensus-contrib-http-util;0.31.1 from central in [default]\n",
      "\tio.opencensus#opencensus-proto;0.2.0 from central in [default]\n",
      "\tio.perfmark#perfmark-api;0.26.0 from central in [default]\n",
      "\tit.unimi.dsi#fastutil;7.0.12 from central in [default]\n",
      "\tjavax.annotation#javax.annotation-api;1.3.2 from central in [default]\n",
      "\tjoda-time#joda-time;2.8.1 from central in [default]\n",
      "\torg.apache.httpcomponents#httpclient;4.5.13 from central in [default]\n",
      "\torg.apache.httpcomponents#httpcore;4.4.13 from central in [default]\n",
      "\torg.checkerframework#checker-qual;3.31.0 from central in [default]\n",
      "\torg.codehaus.mojo#animal-sniffer-annotations;1.22 from central in [default]\n",
      "\torg.conscrypt#conscrypt-openjdk-uber;2.5.2 from central in [default]\n",
      "\torg.projectlombok#lombok;1.16.8 from central in [default]\n",
      "\torg.rocksdb#rocksdbjni;6.29.5 from central in [default]\n",
      "\torg.threeten#threetenbp;1.6.5 from central in [default]\n",
      "\tsoftware.amazon.ion#ion-java;1.0.2 from central in [default]\n",
      "\t:: evicted modules:\n",
      "\tcommons-logging#commons-logging;1.2 by [commons-logging#commons-logging;1.1.3] in [default]\n",
      "\tcommons-codec#commons-codec;1.11 by [commons-codec#commons-codec;1.15] in [default]\n",
      "\tcom.google.protobuf#protobuf-java-util;3.0.0-beta-3 by [com.google.protobuf#protobuf-java-util;3.21.12] in [default]\n",
      "\tcom.google.protobuf#protobuf-java;3.0.0-beta-3 by [com.google.protobuf#protobuf-java;3.21.12] in [default]\n",
      "\tcom.google.code.gson#gson;2.3 by [com.google.code.gson#gson;2.10.1] in [default]\n",
      "\t---------------------------------------------------------------------\n",
      "\t|                  |            modules            ||   artifacts   |\n",
      "\t|       conf       | number| search|dwnlded|evicted|| number|dwnlded|\n",
      "\t---------------------------------------------------------------------\n",
      "\t|      default     |   83  |   0   |   0   |   5   ||   78  |   0   |\n",
      "\t---------------------------------------------------------------------\n",
      ":: retrieving :: org.apache.spark#spark-submit-parent-994cb793-bb56-4b46-ad2f-b20d68529970\n",
      "\tconfs: [default]\n",
      "\t0 artifacts copied, 78 already retrieved (0kB/22ms)\n",
      "24/07/21 10:51:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "Setting default log level to \"WARN\".\n",
      "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n"
     ]
    }
   ],
   "source": [
    "import sparknlp\n",
    "\n",
    "# let's start Spark with Spark NLP with GPU enabled. If you don't have GPUs available remove this parameter.\n",
    "spark = sparknlp.start(gpu=True)\n",
    "print(sparknlp.version())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's use the `loadSavedModel` functon in `AutoGGUFModel`\n",
    "- Most params will be set automatically. They can also be set later after loading the model in `AutoGGUFModel` during runtime, so don't worry about setting them now.\n",
    "- `loadSavedModel` accepts two params, first is the path to the exported model. The second is the SparkSession that is `spark` variable we previously started via `sparknlp.start()`\n",
    "- NOTE: `loadSavedModel` accepts local paths in addition to distributed file systems such as `HDFS`, `S3`, `DBFS`, etc. This feature was introduced in Spark NLP 4.2.2 release. Keep in mind the best and recommended way to move/share/reuse Spark NLP models is to use `write.save` so you can use `.load()` from any file systems natively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracted 'libllama.so' to '/tmp/libllama.so'\n",
      "Extracted 'libjllama.so' to '/tmp/libjllama.so'\n"
     ]
    }
   ],
   "source": [
    "from sparknlp.annotator import *\n",
    "\n",
    "autoGGUFModel = (\n",
    "    AutoGGUFModel.loadSavedModel(EXPORT_PATH, spark)\n",
    "    .setInputCols(\"document\")\n",
    "    .setOutputCol(\"completions\")\n",
    "    .setBatchSize(4)\n",
    "    .setNPredict(20)\n",
    "    .setNGpuLayers(99)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Let's save it on disk so it is easier to be moved around and also be used later via `.load` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "autoGGUFModel.write().overwrite().save(f\"Phi-3.5-mini-instruct-Q4_K_M_spark_nlp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's clean up stuff we don't need anymore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf {EXPORT_PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Awesome  😎 !\n",
    "\n",
    "This is your GGUF model from loaded and saved by Spark NLP 🚀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 2337168\n",
      "drwxr-xr-x 2 ducha ducha       4096 Jul 21 16:24 metadata\n",
      "-rwxrwxr-x 1 ducha ducha 2393231072 Jul 21 16:24 Phi-3.5-mini-instruct-Q4_K_M.gguf\n"
     ]
    }
   ],
   "source": [
    "! ls -l Phi-3.5-mini-instruct-Q4_K_M_spark_nlp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we can use it on other machines, clusters, or any place you wish to use your new and shiny GGUF model 😊"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] build info build=3008 commit=\"1d8fca72\"\n",
      "[INFO] system info n_threads=6 n_threads_batch=-1 total_threads=6 system_info=\"AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | \"\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "llama_model_loader: loaded meta data with 24 key-value pairs and 195 tensors from /tmp/spark-bbad4f64-91a7-4b6e-8242-7f91e6abca54/userFiles-f7d4e4e9-c02d-46e4-81b5-bf5a26d70930/Phi-3.5-mini-instruct-Q4_K_M.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = phi3\n",
      "llama_model_loader: - kv   1:                               general.name str              = Phi3\n",
      "llama_model_loader: - kv   2:                        phi3.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                      phi3.embedding_length u32              = 3072\n",
      "llama_model_loader: - kv   4:                   phi3.feed_forward_length u32              = 8192\n",
      "llama_model_loader: - kv   5:                           phi3.block_count u32              = 32\n",
      "llama_model_loader: - kv   6:                  phi3.attention.head_count u32              = 32\n",
      "llama_model_loader: - kv   7:               phi3.attention.head_count_kv u32              = 32\n",
      "llama_model_loader: - kv   8:      phi3.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv   9:                  phi3.rope.dimension_count u32              = 96\n",
      "llama_model_loader: - kv  10:                          general.file_type u32              = 15\n",
      "llama_model_loader: - kv  11:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  12:                         tokenizer.ggml.pre str              = default\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32064]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32064]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32064]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 32000\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32              = 32000\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  21:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  22:                    tokenizer.chat_template str              = {{ bos_token }}{% for message in mess...\n",
      "llama_model_loader: - kv  23:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   65 tensors\n",
      "llama_model_loader: - type q4_K:   81 tensors\n",
      "llama_model_loader: - type q5_K:   32 tensors\n",
      "llama_model_loader: - type q6_K:   17 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 323/32064 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = phi3\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32064\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 3072\n",
      "llm_load_print_meta: n_head           = 32\n",
      "llm_load_print_meta: n_head_kv        = 32\n",
      "llm_load_print_meta: n_layer          = 32\n",
      "llm_load_print_meta: n_rot            = 96\n",
      "llm_load_print_meta: n_embd_head_k    = 96\n",
      "llm_load_print_meta: n_embd_head_v    = 96\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: n_embd_k_gqa     = 3072\n",
      "llm_load_print_meta: n_embd_v_gqa     = 3072\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: f_logit_scale    = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 8192\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: causal attn      = 1\n",
      "llm_load_print_meta: pooling type     = 0\n",
      "llm_load_print_meta: rope type        = 2\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: ssm_d_conv       = 0\n",
      "llm_load_print_meta: ssm_d_inner      = 0\n",
      "llm_load_print_meta: ssm_d_state      = 0\n",
      "llm_load_print_meta: ssm_dt_rank      = 0\n",
      "llm_load_print_meta: model type       = 3B\n",
      "llm_load_print_meta: model ftype      = Q4_K - Medium\n",
      "llm_load_print_meta: model params     = 3.82 B\n",
      "llm_load_print_meta: model size       = 2.23 GiB (5.01 BPW) \n",
      "llm_load_print_meta: general.name     = Phi3\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 32000 '<|endoftext|>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: PAD token        = 32000 '<|endoftext|>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_print_meta: EOT token        = 32007 '<|end|>'\n",
      "ggml_cuda_init: failed to initialize CUDA: unknown error\n",
      "llm_load_tensors: ggml ctx size =    0.11 MiB\n",
      "llm_load_tensors: offloading 32 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 33/33 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =  2281.66 MiB\n",
      "...........................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 512\n",
      "llama_new_context_with_model: n_batch    = 512\n",
      "llama_new_context_with_model: n_ubatch   = 512\n",
      "llama_new_context_with_model: flash_attn = 0\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "ggml_cuda_host_malloc: failed to allocate 192.00 MiB of pinned memory: unknown error\n",
      "llama_kv_cache_init:        CPU KV buffer size =   192.00 MiB\n",
      "llama_new_context_with_model: KV self size  =  192.00 MiB, K (f16):   96.00 MiB, V (f16):   96.00 MiB\n",
      "ggml_cuda_host_malloc: failed to allocate 0.61 MiB of pinned memory: unknown error\n",
      "llama_new_context_with_model:        CPU  output buffer size =     0.61 MiB\n",
      "ggml_cuda_host_malloc: failed to allocate 83.01 MiB of pinned memory: unknown error\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =    83.01 MiB\n",
      "llama_new_context_with_model: graph nodes  = 1286\n",
      "llama_new_context_with_model: graph splits = 1\n",
      "[Stage 5:>                                                          (0 + 1) / 1]\r"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[INFO] initializing slots n_slots=4\n",
      "[INFO] new slot id_slot=0 n_ctx_slot=128\n",
      "[INFO] new slot id_slot=1 n_ctx_slot=128\n",
      "[INFO] new slot id_slot=2 n_ctx_slot=128\n",
      "[INFO] new slot id_slot=3 n_ctx_slot=128\n",
      "[INFO] model loaded\n",
      "[INFO] chat template chat_example=\"<|system|>\\nYou are a helpful assistant<|end|>\\n<|user|>\\nHello<|end|>\\n<|assistant|>\\nHi there<|end|>\\n<|user|>\\nHow are you?<|end|>\\n<|assistant|>\\n\" built_in=true\n",
      "[INFO] all slots are idle\n",
      "[INFO] slot is processing task id_slot=0 id_task=0\n",
      "[INFO] kv cache rm [p0, end) id_slot=0 id_task=0 p0=0\n",
      "[INFO] prompt eval time     =     318.87 ms /     5 tokens (   63.77 ms per token,    15.68 tokens per second) id_slot=0 id_task=0 t_prompt_processing=318.873 n_prompt_tokens_processed=5 t_token=63.7746 n_tokens_second=15.680223788153905\n",
      "[INFO] generation eval time =    4136.03 ms /    20 runs   (  206.80 ms per token,     4.84 tokens per second) id_slot=0 id_task=0 t_token_generation=4136.032 n_decoded=20 t_token=206.8016 n_tokens_second=4.835552529574239\n",
      "[INFO]           total time =    4454.90 ms id_slot=0 id_task=0 t_prompt_processing=318.873 t_token_generation=4136.032 t_total=4454.905\n",
      "[INFO] slot released id_slot=0 id_task=0 n_ctx=512 n_past=24 n_system_tokens=0 n_cache_tokens=0 truncated=false\n",
      "[INFO] all slots are idle\n",
      "+------------------------------------------------------------------------------------------------------------+\n",
      "|completions                                                                                                 |\n",
      "+------------------------------------------------------------------------------------------------------------+\n",
      "|[{document, 0, 78, 384,000 kilometers away from the Earth. Use scientific notation to express this, {}, []}]|\n",
      "+------------------------------------------------------------------------------------------------------------+\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "import sparknlp\n",
    "from sparknlp.base import *\n",
    "from sparknlp.annotator import *\n",
    "from pyspark.ml import Pipeline\n",
    "\n",
    "document_assembler = DocumentAssembler().setInputCol(\"text\").setOutputCol(\"document\")\n",
    "\n",
    "auto_gguf_model = AutoGGUFModel.load(\"Phi-3.5-mini-instruct-Q4_K_M_spark_nlp\")\n",
    "\n",
    "pipeline = Pipeline().setStages([document_assembler, auto_gguf_model])\n",
    "\n",
    "data = spark.createDataFrame([[\"The moon is \"]]).toDF(\"text\")\n",
    "\n",
    "result = pipeline.fit(data).transform(data)\n",
    "result.select(\"completions\").show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! You can now go wild and use hundreds of GGUF models from HuggingFace 🤗 in Spark NLP 🚀\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}