sentence_embedding.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "<center>\n",
        "    <p style=\"text-align:center\">\n",
        "        <img alt=\"BentoML logo\" src=\"https://raw.githubusercontent.com/bentoml/BentoML/main/docs/source/_static/img/bentoml-logo-black.png\" width=\"200\"/>\n",
        "        </br>\n",
        "        <a href=\"https://github.com/bentoml/\">GitHub</a>\n",
        "        |\n",
        "        <a href=\"https://l.bentoml.com/join-openllm-discord\">Community</a>\n",
        "    </p>\n",
        "</center>\n",
        "<h1 align=\"center\">Sentence Embedding with BentoML</h1>\n",
        "\n",
        "BentoML is a framework for building reliable, scalable, and cost-efficient AI applications. It comes with everything you need for model serving, application packaging, and production deployment.\n",
        "\n",
        "This is a live demo for [sentence-embedding-bento](https://github.com/bentoml/sentence-embedding-bento). In this tutorial, you will learn the following:\n",
        "- Set up your environment to work with BentoML\n",
        "- Serve a REST API server for generating text embeddings with just one-line command\n",
        "- Explore different ways to interact with the server\n",
        "- Build the bentos for deployment\n",
        "- Production Deployment"
      ],
      "metadata": {
        "id": "jXa4vnd9dVqD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Set up"
      ],
      "metadata": {
        "id": "tGag3VzkiFlc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before diving into this project, let's ensure our environment has everything in place."
      ],
      "metadata": {
        "id": "g0TC_D9qiLc4"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "-nr4-N4Oa4ex",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0019bdb0-e759-47b9-aea9-987d38eae903"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "-= Updating repo =-\n",
            "Already up to date.\n",
            "Installing dependencies...\n",
            "Done!\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "\n",
        "UPDATE_REPO = True  #@param {type:\"boolean\"}\n",
        "PROJECT_NAME = 'sentence-embedding-bento'\n",
        "\n",
        "if not PROJECT_NAME in os.getcwd():\n",
        "  ![ ! -d $PROJECT_NAME ] && echo -= Initial setup =- && git clone https://github.com/bentoml/sentence-embedding-bento.git\n",
        "  %cd $PROJECT_NAME\n",
        "\n",
        "if UPDATE_REPO:\n",
        "  !echo -= Updating repo =- && git pull\n",
        "\n",
        "print(\"Installing dependencies...\")\n",
        "!pip install --upgrade -q --progress-bar off -r requirements.txt\n",
        "print(\"Done!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Serve sentence-embedding!"
      ],
      "metadata": {
        "id": "ZFVarjGVmjWT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "There are 2 possible ways to get the server ready:\n",
        "\n",
        "- attempt to run server in the Colab runtime locally\n",
        "- Or you can choose to launch it outside the colab environment as the resource limits of free Colab are quite small for LLMs."
      ],
      "metadata": {
        "id": "C_9k8ybzs1DD"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Run it in Colab"
      ],
      "metadata": {
        "id": "hgR6XIJGu1Gm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Fistly, let's download a sentence embedding model using bentoml sdk.\n",
        "This will save `all-MiniLM-L6-v2` in your local BentoML model store.\n",
        "\n",
        "💁 Here, you can try any other embedding models in https://huggingface.co/models?library=sentence-transformers just change the model variable in this code block"
      ],
      "metadata": {
        "id": "YhzF-5jcmvyZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import bentoml\n",
        "from transformers import AutoTokenizer, AutoModel\n",
        "\n",
        "model_id=\"sentence-transformers/all-MiniLM-L6-v2\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
        "model = AutoModel.from_pretrained(model_id)\n",
        "\n",
        "bentoml.transformers.save_model(\"all-MiniLM-L6-v2\", model)\n",
        "bentoml.transformers.save_model(\"all-MiniLM-L6-v2-tokenizer\", tokenizer)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x9U9049GmDZn",
        "outputId": "68667db0-4bf2-49c5-cfee-8dbecce983d5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:bentoml._internal.tag:Converting 'all-MiniLM-L6-v2' to lowercase: 'all-minilm-l6-v2'.\n",
            "WARNING:bentoml._internal.tag:Converting 'all-MiniLM-L6-v2-tokenizer' to lowercase: 'all-minilm-l6-v2-tokenizer'.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Model(tag=\"all-minilm-l6-v2-tokenizer:ofib35tmto5keasc\", path=\"/root/bentoml/models/all-minilm-l6-v2-tokenizer/ofib35tmto5keasc/\")"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "serving it is straightforward with BentoML. With just a single command, you're good to go:\n",
        "\n",
        "> 👀 bentoml will start a server difinted in `service.py` listening 3000 port as deafult. And to unblock the following steps, we will run it in background via `nohup`"
      ],
      "metadata": {
        "id": "ePXKoKXYpqj3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nohup bentoml serve > bentoml.log 2>&1 &\n",
        "SERVER_URL = \"http://localhost:3000\""
      ],
      "metadata": {
        "id": "aC3GXFktpqTn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Run it outside"
      ],
      "metadata": {
        "id": "wUqC8-cTu7ZX"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, ensure you have Docker installed and running in your server.\n",
        "\n",
        "Launch the embedding service with the prebuilt image by the following command:\n",
        "\n",
        "```\n",
        "docker run --rm -p 3000:3000 ghcr.io/bentoml/sentence-embedding-bento:latest\n",
        "```"
      ],
      "metadata": {
        "id": "gj3CiZ7evMEM"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To run model inference with GPU, install the NVIDIA Container Toolkit and use the GPU-enabled docker image instead:\n",
        "\n",
        "```\n",
        "docker run --gpu --rm -p 3000:3000 ghcr.io/bentoml/sentence-embedding-bento-gpu:latest\n",
        "```"
      ],
      "metadata": {
        "id": "d-KXQkJnzEdA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "provide the URL of your own server"
      ],
      "metadata": {
        "id": "7Xx_kXijx7Cz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "SERVER_URL = \"http://xxx.xx\" # @param {type: 'string'}"
      ],
      "metadata": {
        "id": "C0LB48Qkv-gu"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Try sentence-embedding server"
      ],
      "metadata": {
        "id": "DkFWHn4fuARk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Check server status"
      ],
      "metadata": {
        "id": "2AtsJ339yCDC"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Before you interact with the OpenLLM server, it's crucial to ensure that it is up and running. The output of the `curl` command should start with `HTTP/1.1 200 OK`, meaning everything is in order.\n",
        "\n",
        "If it says `curl: (6) Could not resolve host: SERVER_URL`, ensure you have run the setup step.\n",
        "\n",
        "If it says `curl: (7) Failed to connect to localhost...`, then check `./openllm.log` and `./openllm.err`; likely the server has failed to start or is still in the process of starting.\n",
        "\n",
        "If it says `HTTP/1.1 503 Service Unavailable`, the server is still starting and you should wait a bit and retry."
      ],
      "metadata": {
        "id": "Mml9GuQfyI6Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!curl -i {SERVER_URL}/readyz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YH15vz1wyDB8",
        "outputId": "fb4705a4-756e-4d38-dcc0-6700d6850f60"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "HTTP/1.1 200 OK\r\n",
            "\u001b[1mdate\u001b[0m: Tue, 17 Oct 2023 03:58:05 GMT\r\n",
            "\u001b[1mserver\u001b[0m: uvicorn\r\n",
            "\u001b[1mcontent-length\u001b[0m: 1\r\n",
            "\u001b[1mcontent-type\u001b[0m: text/plain; charset=utf-8\r\n",
            "\r\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Web UI"
      ],
      "metadata": {
        "id": "T2Mz9jhhyQiF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "if 'google.colab' in sys.modules:\n",
        "    #using colab proxy URL\n",
        "    from google.colab.output import eval_js\n",
        "    print(\"you are in colab runtime. try it out in %s\" % eval_js(\"google.colab.kernel.proxyPort(3000)\"))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "id": "GzQPQr0pyiwN",
        "outputId": "3386c74c-3177-4655-e6d4-77f0e6ac8601"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "you are in colab runtime. try it out in https://1c7imreyyom-496ff2e9c6d22116-3000-colab.googleusercontent.com/\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Raw HTTP"
      ],
      "metadata": {
        "id": "ZIzLNV0YyOV9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!curl -X POST {SERVER_URL}/encode \\\n",
        "   -H 'Content-Type: application/json' \\\n",
        "   -d '[\"hello world, how are you?\", \"I love fried chicken sandwiches!\"]'"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "51bBD9SYyyps",
        "outputId": "9bde8d2f-b234-4e4c-e3b3-477abf72ff00"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[0.17826364934444427, 0.10717688500881195, 0.43587279319763184, 0.20607613027095795, 0.0022702317219227552, -0.6842638254165649, 0.28179970383644104, 0.004625807050615549, -0.5464844703674316, 0.12071722000837326, 0.008337091654539108, 0.05009095370769501, -0.009813488461077213, -0.09429076313972473, 0.11632076650857925, -0.311547189950943, 0.31155329942703247, -0.5299899578094482, -0.6968725323677063, 0.11798126250505447, -0.13786333799362183, 0.2133282721042633, 0.3239438831806183, 0.33048930764198303, -0.3040446937084198, -0.14464575052261353, 0.059592656791210175, 0.191707581281662, -0.014646864496171474, -0.41965341567993164, -0.27630409598350525, 0.2672696113586426, 0.1408645063638687, -0.12229768186807632, -0.130007803440094, 0.24715524911880493, -0.05008126050233841, -0.730891764163971, -0.20369082689285278, 0.015536967664957047, 0.35080134868621826, -0.15579760074615479, -0.08321177959442139, -0.15525977313518524, 0.30729877948760986, -0.26019930839538574, 0.03653553128242493, 0.3854633867740631, 0.4132363796234131, -0.08162017911672592, -0.44153887033462524, 0.0008560485439375043, -0.07123652845621109, 0.25672730803489685, 0.16862305998802185, 0.14641372859477997, -0.2060910165309906, 0.19015951454639435, 0.20786279439926147, -0.0709749162197113, -0.052141688764095306, 0.02121799811720848, -0.22805415093898773, 0.394540935754776, 0.4269756078720093, -0.1407652199268341, -0.27914196252822876, -0.11049704998731613, -0.26477956771850586, -0.27522408962249756, -0.29539453983306885, -0.17177817225456238, 0.1924571394920349, 0.25308459997177124, -0.33845895528793335, -0.28658172488212585, 0.11743499338626862, -0.22076286375522614, 0.2902612090110779, 0.2626740038394928, 0.46371036767959595, -0.4117473065853119, 0.033875856548547745, 0.18003523349761963, -0.11763012409210205, -0.22985699772834778, 0.1148892268538475, 0.19361336529254913, -0.1839362233877182, -0.008095749653875828, -0.5880296230316162, 0.25308316946029663, 0.1104242205619812, 0.11319127678871155, -0.04364302381873131, -0.13737867772579193, 0.5608010292053223, -0.1753585934638977, -0.6063591837882996, 0.7673570513725281, 0.15010450780391693, 0.4512275159358978, 0.2441284954547882, -0.018115252256393433, 0.23608091473579407, 0.3296840190887451, -0.4553163945674896, 0.48796477913856506, -0.22386401891708374, -0.27727407217025757, -0.0824582427740097, -0.14042232930660248, -0.008247477002441883, -0.14360852539539337, 0.43033701181411743, -0.324866384267807, 0.07800650596618652, 0.24951304495334625, 0.03488575294613838, 0.14186006784439087, 0.11807510256767273, 0.0226447731256485, 0.16490712761878967, 0.2218223214149475, -0.13924692571163177, 0.3446325361728668, 0.3064426779747009, -3.4481240112938734e-32, 0.6148366928100586, 0.06145477294921875, 0.43179380893707275, 0.8580018281936646, -0.22339995205402374, -0.009902936406433582, -0.3014930784702301, -0.09736720472574234, 0.24897639453411102, 0.08449128270149231, 0.17891079187393188, 0.10861407220363617, -0.07662317156791687, 0.05408107489347458, -0.2551764249801636, 0.09250269830226898, -0.1704215556383133, 0.041532013565301895, -0.06053883582353592, 0.8867390155792236, -0.1971520483493805, -0.20225580036640167, -0.09244651347398758, 0.18121522665023804, 0.20335692167282104, 0.14778752624988556, -0.03186438977718353, -0.5249800086021423, 0.18970364332199097, -0.028027674183249474, -0.09818974137306213, -0.1137971356511116, 0.1553037464618683, -0.1991460621356964, 0.13653896749019623, -0.11642777919769287, 0.0561344213783741, -0.06478414684534073, -0.17240512371063232, -0.05545291677117348, -0.2482791543006897, 0.2027018815279007, 0.5537810921669006, -0.12851102650165558, -0.044631026685237885, -0.22201544046401978, -0.07548599690198898, 0.13598662614822388, -0.07400736212730408, 0.18773505091667175, -0.4177033007144928, 0.05531671270728111, -0.29180124402046204, 0.20217137038707733, -0.2216738611459732, -0.16560475528240204, 0.08277516812086105, -0.038556694984436035, -0.05668704956769943, 0.5011939406394958, 0.10003823041915894, 0.5966665744781494, -0.06697957962751389, -0.2639425992965698, -0.24608442187309265, -0.10982677340507507, -0.11067286878824234, -0.06251247972249985, 0.4652726948261261, -0.3625170588493347, 0.15066030621528625, -0.1378784030675888, -0.023162206634879112, 0.47806528210639954, 0.18810786306858063, 0.2405422478914261, 0.18638910353183746, -0.2654085159301758, -0.04689842090010643, -0.390047162771225, 0.2425118088722229, 0.558188796043396, -0.07080674916505814, -0.08924620598554611, 0.28802043199539185, -0.20000068843364716, -0.21579858660697937, -0.5169704556465149, -0.10229694843292236, 0.18918970227241516, -0.21445894241333008, 0.2631453275680542, 0.28468769788742065, -0.1429489254951477, -0.4302240312099457, 2.653513046151436e-32, 0.8091317415237427, -0.07818514853715897, -0.19718892872333527, -0.20313721895217896, -0.5264208316802979, -0.24043114483356476, -0.40580877661705017, 0.6467493772506714, -0.299021452665329, 0.20337261259555817, 0.4528118669986725, 0.08684632182121277, 0.39349091053009033, 0.02541695535182953, 0.28622081875801086, -0.20081239938735962, 0.47671598196029663, 0.09953504055738449, -0.04917321726679802, -0.07230781763792038, -0.13042065501213074, 0.20711643993854523, -0.19826169312000275, 0.2732505202293396, -0.13719932734966278, 0.009757740423083305, 0.12573006749153137, 0.2239433377981186, -0.20907214283943176, 0.0034007462672889233, -0.06244116276502609, -0.01941179297864437, -0.24182187020778656, 0.34469425678253174, 0.1585562527179718, 0.23113209009170532, -0.3385807275772095, -0.34566035866737366, -0.2842819094657898, -0.43790939450263977, -0.5155289173126221, 0.23784486949443817, -0.05014569312334061, 0.3100103437900543, 0.15845265984535217, -0.14953574538230896, -0.10208678245544434, 0.09739306569099426, -0.01618364080786705, 0.03801843151450157, -0.11708278208971024, -0.005874725058674812, 0.0967208668589592, -0.0819697305560112, -0.04070259630680084, -0.045312777161598206, -0.3849761486053467, -0.2708151340484619, 0.14753100275993347, -0.13468195497989655, -0.1926807463169098, 0.022411644458770752, -0.0588347390294075, 0.23083345592021942, 0.2688252925872803, -0.16222161054611206, 0.05573207885026932, 0.15141089260578156, -0.037714723497629166, -0.3822818398475647, -0.3737209439277649, -0.10144167393445969, 0.01882646232843399, -0.16413719952106476, 0.05170544981956482, 0.2821301221847534, 0.057835742831230164, -0.09260152280330658, -0.08665793389081955, 0.09032668173313141, -0.0578727051615715, 0.12562304735183716, 0.21617631614208221, -0.4400189220905304, -0.39458414912223816, 0.03262708708643913, 0.3623981177806854, 0.29658210277557373, 0.07718423008918762, 0.13065366446971893, 0.05521433427929878, 0.18799011409282684, -0.12104197591543198, 0.14802837371826172, 0.014423713088035583, -9.182370064308998e-08, -0.18756315112113953, 0.1503714919090271, -0.2599750757217407, 0.17588067054748535, -0.05478242412209511, 0.20238259434700012, -0.33132606744766235, -0.16331954300403595, -0.3021249771118164, -0.12507344782352448, 0.29897934198379517, 0.2281685471534729, -0.4936915934085846, -0.38986560702323914, 0.15852157771587372, 0.14924442768096924, 0.26110270619392395, 0.3541626036167145, -0.2866406738758087, -0.18082892894744873, 0.18440577387809753, 0.06864664703607559, -0.015154872089624405, 0.182754248380661, 0.10129263997077942, -0.1707797646522522, 0.09843376278877258, 0.008030395023524761, -0.12101960182189941, 0.03421200066804886, 0.19477540254592896, 0.6956133842468262, -0.22804440557956696, 0.2589627504348755, -0.2453516572713852, -0.18902018666267395, 0.14606402814388275, 0.1761932373046875, 0.24694103002548218, 0.04679529368877411, -0.22620360553264618, 0.20434227585792542, -0.33256232738494873, -0.3098405599594116, 0.13119840621948242, -0.009396612644195557, 0.3964439928531647, -0.39517050981521606, -0.0073310863226652145, -0.08703423291444778, -0.32052409648895264, 0.043888285756111145, 0.32799679040908813, 0.09408237785100937, 0.3456147313117981, 0.3680780529975891, -0.04061431065201759, 0.3680095076560974, -0.18747305870056152, 0.1850908398628235, 0.045668940991163254, -0.21905332803726196, -0.30004215240478516, 0.05176933482289314], [-0.6013821363449097, -0.4426668584346771, -0.03185832127928734, 0.0797944962978363, -0.22276195883750916, -0.1582765430212021, 0.21857774257659912, -0.24650315940380096, 0.5547654628753662, -0.041287947446107864, -0.20635506510734558, 0.16584914922714233, 0.07454830408096313, 0.11757205426692963, 0.17802134156227112, -0.33590900897979736, 0.5717570781707764, -0.14571097493171692, -0.14342591166496277, -0.2982148230075836, -0.7997370958328247, 0.06874645501375198, 0.18645736575126648, -0.23166769742965698, -0.35371237993240356, 0.2735288143157959, 0.16178187727928162, -0.2068539261817932, -0.7205393314361572, -0.1590862274169922, -0.03775022551417351, 0.07635239511728287, -0.46237775683403015, 0.2470177859067917, 0.09665495157241821, 0.06160593032836914, 0.49579569697380066, -0.035568706691265106, 0.30772361159324646, 0.1639813482761383, 0.033940453082323074, 0.02909659408032894, 0.6746116876602173, -0.3012509346008301, 0.3819841146469116, 0.2214897871017456, -0.23660506308078766, 0.059338927268981934, 0.210950568318367, -0.020133353769779205, 0.048306629061698914, 0.28686922788619995, 0.10072191804647446, -0.29736343026161194, 0.28804507851600647, 0.5168310403823853, -0.48907437920570374, -0.4674302637577057, -0.625738263130188, 0.08452290296554565, -0.20107430219650269, 0.35391664505004883, -0.26836729049682617, 0.12637239694595337, -0.158472940325737, -0.6614686846733093, 0.0031038597226142883, 0.3086807131767273, -0.046827495098114014, 0.3682674765586853, -0.16177499294281006, 0.3455774784088135, 0.3522104024887085, 0.4333147704601288, -0.180559903383255, -0.06312086433172226, 0.4129072427749634, -0.612725555896759, 0.1451358199119568, -0.2249397486448288, -0.0677947923541069, -0.48320019245147705, -0.5812877416610718, 0.3454322814941406, -0.010465123690664768, -0.39132314920425415, -0.0018835701048374176, 0.18953871726989746, -0.18321944773197174, -0.08589653670787811, 0.36360809206962585, -0.1516115516424179, -0.14849857985973358, 0.22128626704216003, -0.3431466817855835, -0.7424198389053345, -0.08186788856983185, -0.3447788953781128, -0.8490884304046631, 0.373334139585495, -0.05291137844324112, 0.09324418753385544, -0.09081850200891495, -0.4101749062538147, -0.0029603647999465466, -0.0320146381855011, 0.0064816903322935104, 0.4109884202480316, 0.025127708911895752, -0.0571446418762207, 0.20303523540496826, 0.1516437530517578, 0.03777402639389038, -0.3593178391456604, -0.39127451181411743, -0.1752987504005432, 0.6827347278594971, -0.4030637741088867, -0.02278662472963333, 0.288998544216156, -0.16187414526939392, 0.420274555683136, -0.04715224355459213, 0.37968745827674866, -0.5399176478385925, -0.05632304400205612, 0.087287038564682, -2.111273959038026e-32, -0.5065337419509888, 0.01435488648712635, 0.4991902709007263, -0.11629745364189148, 0.29984235763549805, -0.2031802535057068, 0.1927855908870697, -0.015796061605215073, -0.08903102576732635, 0.32248470187187195, -0.3009031414985657, -0.20157824456691742, 0.061881329864263535, 0.45919328927993774, -0.20015259087085724, -0.19004148244857788, -0.2034081369638443, -0.13367366790771484, -0.03467497229576111, -0.35716789960861206, -0.2656251788139343, 0.47368133068084717, 0.7388561367988586, 0.45117825269699097, 0.22961117327213287, -0.5576400756835938, 0.1994602084159851, -0.506030797958374, 0.01699964702129364, 0.3564862310886383, 0.17887234687805176, 0.05078595131635666, -0.14147435128688812, -0.03639480844140053, -0.4545609951019287, 0.303094744682312, -0.4486720561981201, -0.19482888281345367, -0.012607062235474586, 0.6831691265106201, 0.35329174995422363, -0.3657107651233673, -0.14151591062545776, 0.3831331133842468, -0.010226206853985786, 0.07564650475978851, 0.1388397365808487, 0.11625204980373383, 0.2943893373012543, -0.45530518889427185, -0.1081472858786583, -0.37678757309913635, 0.29104816913604736, 0.1446116864681244, 0.16112615168094635, -0.18739750981330872, 0.28472456336021423, 0.13876241445541382, 0.040362399071455, -0.05483608320355415, -0.3808206021785736, 0.2158070057630539, -0.27159035205841064, -0.4464280605316162, -0.008067324757575989, 0.12398755550384521, -0.059051819145679474, -0.3584176301956177, -0.36261510848999023, -0.15169867873191833, 0.6590638756752014, -0.26747262477874756, 0.1218428686261177, -0.30050501227378845, -0.018749035894870758, -0.09435024112462997, -0.146498441696167, 0.1464223861694336, -0.018201863393187523, -0.3250860869884491, 0.9641185402870178, 0.17878344655036926, 0.10547620058059692, -0.02221091277897358, 0.1928872913122177, 0.42316970229148865, -0.36011630296707153, -0.31512734293937683, 0.4338463246822357, 0.08247841894626617, -0.1149265468120575, -0.704893171787262, 0.7066605687141418, -0.33676451444625854, -0.47484880685806274, 1.509870918816586e-32, -0.04592227190732956, -0.2583044171333313, -0.13111257553100586, 0.3005527853965759, -0.10143090039491653, -0.6875657439231873, -0.38267961144447327, 0.35335829854011536, -0.07759210467338562, -0.08377973735332489, -0.15582238137722015, -0.25096866488456726, -0.2895587980747223, -0.1386479139328003, -0.464428186416626, 0.43103983998298645, 0.05223787575960159, 0.30511778593063354, 0.28247925639152527, -0.20954276621341705, -0.28648635745048523, -0.2625424265861511, 0.29366663098335266, 0.7988197207450867, -0.34461116790771484, 0.49969545006752014, 0.3880751132965088, 0.6290146708488464, -0.17696142196655273, -0.11301599442958832, -0.012547105550765991, -0.2092524617910385, -0.13919751346111298, -0.5180066823959351, -0.06931888312101364, 0.3249701261520386, -0.01954900473356247, -0.12744559347629547, 0.011033019050955772, -0.1569511443376541, -0.21878917515277863, 0.4620685875415802, 0.7922095060348511, 0.7282516956329346, 0.14032447338104248, -0.3093285858631134, -0.058362118899822235, -0.32323023676872253, -0.6023686528205872, -0.14509862661361694, -0.5708709359169006, -0.571662425994873, 0.49689701199531555, -0.4504883587360382, -0.08340024948120117, 0.15804384648799896, 0.4257451593875885, 0.2543014585971832, 0.17319750785827637, -0.11860306560993195, -0.34347644448280334, 0.2051553577184677, 0.3075208365917206, 0.5179471969604492, 1.0285793542861938, -0.13395661115646362, -0.06961730867624283, -0.4514404237270355, -0.07057097554206848, 0.017634497955441475, -0.3171992599964142, 0.021795164793729782, -0.23133906722068787, -0.14646217226982117, 0.1265903115272522, -0.26141539216041565, 0.0841958150267601, 0.264558881521225, -0.0397968590259552, 0.5047286152839661, -0.06539630144834518, -0.3846682012081146, -0.18194925785064697, 0.4620359241962433, -0.06384829431772232, 0.5666268467903137, 0.217421293258667, 0.2838976979255676, 0.01779772900044918, 0.027630086988210678, 0.10163351148366928, -0.1957070678472519, 0.0034237727522850037, 0.13541555404663086, 0.24668186902999878, -9.38120408022769e-08, 0.560617208480835, 0.09540645033121109, 0.04399399086833, 0.32380110025405884, -0.04737035185098648, -0.012602980248630047, -0.07961125671863556, -0.3383011519908905, 0.21834872663021088, -0.307931125164032, -0.11109389364719391, 0.597459614276886, 0.008485771715641022, 0.536581814289093, 0.12671473622322083, 0.2653087079524994, 0.24204836785793304, 0.09716365486383438, 0.056920669972896576, 0.25597184896469116, 0.1566479206085205, 0.5185607075691223, 0.2528800666332245, 0.125690296292305, 0.3183029890060425, -0.14693623781204224, -0.17603455483913422, -0.08144772052764893, 0.4023704528808594, -0.08613228052854538, -0.22388973832130432, 0.09982303529977798, -0.03586190193891525, 0.5011980533599854, 0.26569703221321106, -0.3852332830429077, -0.089141346514225, -0.10406477749347687, -0.023020002990961075, -0.4567514955997467, -0.23805685341358185, -0.22049842774868011, -0.09573730081319809, -0.3101489543914795, -0.3833152651786804, -0.1300622522830963, 0.020794488489627838, 0.4045664072036743, -0.22646011412143707, 0.08816003799438477, -0.1612766534090042, 0.34197548031806946, 0.3262881338596344, 0.2557624876499176, 0.05553237348794937, -0.07225728780031204, 0.12242799252271652, -0.2710324823856354, 0.5506651401519775, 0.18830344080924988, 0.17787937819957733, 0.4581785500049591, -0.4402627944946289, -0.005609060171991587]]"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### BentoML client"
      ],
      "metadata": {
        "id": "bxrrCI-5ybcU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from bentoml.client import Client\n",
        "import nest_asyncio\n",
        "nest_asyncio.apply()\n",
        "\n",
        "client = Client.from_url(SERVER_URL)\n",
        "\n",
        "samples = [\n",
        "  \"The dinner was great!\",\n",
        "  \"The weather is great today!\",\n",
        "  \"I love fried chicken sandwiches!\"\n",
        "]\n",
        "print(client.encode(samples))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "e68J0UAezZR7",
        "outputId": "2c6c1ee0-1362-4a0e-d459-217e271bf216"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:bentoml._internal.io_descriptors.json:BentoML does not support loading pydantic models from URLs; output will be a normal dictionary.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[[-0.20782252  0.57161289 -0.12791678 ...  0.51041508 -0.54378605\n",
            "  -0.05389049]\n",
            " [-0.11160703  0.43717682  0.80714953 ... -0.07692268 -0.70888156\n",
            "   0.53045732]\n",
            " [-0.6013819  -0.44266695 -0.03185841 ...  0.45817858 -0.44026256\n",
            "  -0.00560874]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Build Bento"
      ],
      "metadata": {
        "id": "pp-Cwco0z3Sg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Bento is the standardize distribution format, which is supported by an array of downstream deployment tools provided in the BentoML eco-system. It captures your service code, models, and configurations in one place, version control it automatically, and ensures reproducibility across yoru development and production environments. Learn more in [BentoML Documentation](https://docs.bentoml.com/en/latest/concepts/bento.html)"
      ],
      "metadata": {
        "id": "16ZvtJMjz50T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!bentoml build -f bentofile.yaml"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V_UohwX50GY7",
        "outputId": "b04a592b-2794-4bc0-8f0d-edc81ab49154"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Converting 'all-MiniLM-L6-v2' to lowercase: 'all-minilm-l6-v2'.\n",
            "Converting 'all-MiniLM-L6-v2-tokenizer' to lowercase: 'all-minilm-l6-v2-tokenizer'.\n",
            "'labels' should be a dict[str, str] and enforced by BentoML. Converting all values to string.\n",
            "\n",
            "██████╗ ███████╗███╗   ██╗████████╗ ██████╗ ███╗   ███╗██╗\n",
            "██╔══██╗██╔════╝████╗  ██║╚══██╔══╝██╔═══██╗████╗ ████║██║\n",
            "██████╔╝█████╗  ██╔██╗ ██║   ██║   ██║   ██║██╔████╔██║██║\n",
            "██╔══██╗██╔══╝  ██║╚██╗██║   ██║   ██║   ██║██║╚██╔╝██║██║\n",
            "██████╔╝███████╗██║ ╚████║   ██║   ╚██████╔╝██║ ╚═╝ ██║███████╗\n",
            "╚═════╝ ╚══════╝╚═╝  ╚═══╝   ╚═╝    ╚═════╝ ╚═╝     ╚═╝╚══════╝\n",
            "\n",
            "\u001b[32mSuccessfully built Bento(tag=\"sentence-embedding-svc:wt2ymutmukn2iasc\").\u001b[0m\n",
            "\u001b[34m\n",
            "Possible next steps:\n",
            "\n",
            " * Containerize your Bento with `bentoml containerize`:\n",
            "    $ bentoml containerize sentence-embedding-svc:wt2ymutmukn2iasc  [or bentoml build --containerize]\u001b[0m\n",
            "\u001b[34m\n",
            " * Push to BentoCloud with `bentoml push`:\n",
            "    $ bentoml push sentence-embedding-svc:wt2ymutmukn2iasc [or bentoml build --push]\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Production Deployment"
      ],
      "metadata": {
        "id": "VUcrt1ed0e8l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "BentoML provides a number of deployment options. The easiest way to set up a production-ready endpoint of your text embedding service is via BentoCloud, the serverless cloud platform built for BentoML, by the BentoML team.\n",
        "\n",
        "Next steps:\n",
        "\n",
        "1. Sign up for a BentoCloud account [here](https://www.bentoml.com/).\n",
        "2. Get an API Token, see instructions [here](https://docs.bentoml.com/en/latest/bentocloud/getting-started/ship.html#acquiring-an-api-token).\n",
        "3. Push your Bento to BentoCloud: `bentoml push sentence-embedding-svc:latest`\n",
        "4. Deploy via Web UI, see [Deploying on BentoCloud](https://docs.bentoml.com/en/latest/bentocloud/getting-started/ship.html#deploying-your-bento)"
      ],
      "metadata": {
        "id": "uogrvMuG05T_"
      }
    }
  ]
}