large-scale-recsys/01-Large-Scale-Recsys-Deploy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f771572a",
   "metadata": {},
   "source": [
    "# 1 Large-Scale Recommender Models\n",
    "\n",
    "The last notebook that shows how to handle very large datasets when training models like DLRM for recommendation systems. Large enterprises often have millions of users and items. The entire embedding table of a model may not fit on a single GPU. For this, NVIDIA created the HugeCTR framework.\n",
    "\n",
    "[HugeCTR](https://github.com/NVIDIA-merlin/HugeCTR) is a part of a NVIDIA Merlin framework and adds facilities for distributed training and serving of recommendation models. The notebook detailed here focuses on the deployment and serving of HugeCTR and provides a pre-trained version of DLRM that can be used for the example. \n",
    "\n",
    "In this notebook, we show how to take the last stage of the \"online\" pipeline shown in the previous set of notebooks and demonstrate how to host and infer from the same DLRM model when the embedding table is larger than GPU memory. For the sake of this demonstration, the model is only ~4Gb, however HugeCTR can support embedding tables into the multi-terabyte range across many NVLinked GPUs.\n",
    "\n",
    "While HugeCTR has both support for training and inference for large-scale models, here we focus on the deployment (either locally or cloud-based with the terraform instructions) using the Hierarchical Parameter Server (HPS). The HPS is a tiered-memory system that allows recommender models like DLRM utilize embeddings stored across many compute instances. The diagram for the HPS is show below\n",
    "\n",
    "![img](https://raw.githubusercontent.com/RedisVentures/Redis-Recsys/master/assets/LargeScaleRecsysHugeCTR.png)\n",
    "\n",
    "The below cells show how to\n",
    "1. Generate the Triton and HugeCTR HPS configurations for hosting\n",
    "2. Provide instructions to run Triton with HugeCTR\n",
    "3. Show how to benchmark the model using Perf Analyzer from NVIDIA\n",
    "4. Show how to make HTTPs calls using the Triton Python client\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7683ee9e",
   "metadata": {},
   "source": [
    "## 2. Generate the DLRM Deployment Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "17fecafa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define some data folder to store the model related files\n",
    "# Standard Libraries\n",
    "import os\n",
    "from time import time\n",
    "import re\n",
    "import shutil\n",
    "import glob\n",
    "import warnings\n",
    "\n",
    "DATA_DIR  = \"/model-data/criteo/\"\n",
    "model_folder  = os.path.join(DATA_DIR, \"model\")\n",
    "dlrm_model_repo= os.path.join(model_folder, \"dlrm\")\n",
    "dlrm_version =os.path.join(dlrm_model_repo, \"1\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6db35a79",
   "metadata": {},
   "source": [
    "### Generate Triton configuration for DLRM Deployment "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3b0b5c91",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /model-data/criteo/model/dlrm/config.pbtxt\n"
     ]
    }
   ],
   "source": [
    "%%writefile $dlrm_model_repo/config.pbtxt\n",
    "name: \"dlrm\"\n",
    "backend: \"hugectr\"\n",
    "max_batch_size:64,\n",
    "input [\n",
    "   {\n",
    "    name: \"DES\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ -1 ]\n",
    "  },\n",
    "  {\n",
    "    name: \"CATCOLUMN\"\n",
    "    data_type: TYPE_INT64\n",
    "    dims: [ -1 ]\n",
    "  },\n",
    "  {\n",
    "    name: \"ROWINDEX\"\n",
    "    data_type: TYPE_INT32\n",
    "    dims: [ -1 ]\n",
    "  }\n",
    "]\n",
    "output [\n",
    "  {\n",
    "    name: \"OUTPUT0\"\n",
    "    data_type: TYPE_FP32\n",
    "    dims: [ -1 ]\n",
    "  }\n",
    "]\n",
    "instance_group [\n",
    "  {\n",
    "    count: 1\n",
    "    kind : KIND_GPU\n",
    "    gpus:[0]\n",
    "  }\n",
    "]\n",
    "\n",
    "parameters [\n",
    "  {\n",
    "    key: \"config\"\n",
    "    value: { string_value: \"/model-data/criteo/model/dlrm/1/dlrm.json\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"gpucache\"\n",
    "    value: { string_value: \"true\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"hit_rate_threshold\"\n",
    "    value: { string_value: \"0.5\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"gpucacheper\"\n",
    "    value: { string_value: \"0.1\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"label_dim\"\n",
    "    value: { string_value: \"1\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"slots\"\n",
    "    value: { string_value: \"26\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"cat_feature_num\"\n",
    "    value: { string_value: \"26\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"des_feature_num\"\n",
    "    value: { string_value: \"13\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"max_nnz\"\n",
    "    value: { string_value: \"2\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"embedding_vector_size\"\n",
    "    value: { string_value: \"128\" }\n",
    "  },\n",
    "  {\n",
    "    key: \"embeddingkey_long_type\"\n",
    "    value: { string_value: \"true\" }\n",
    "  }\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8b98e5b",
   "metadata": {},
   "source": [
    "### Generate HPS configuration for DLRM deployment\n",
    "\n",
    "The cell generates the configuration for the HPS. The below cell is setup to use the docker-networked Redis cluster and a GPU cache utilizing only two layers of the HPS system (not SSD).\n",
    "\n",
    "Configuration settings such as partitions, cache rates, and batch sizes matter quite a bit for performance. These settings will vary based on the size and number of the models being deployed, size of the redis cluster, and size of the GPU itself. For more information, see the HugeCTR HPS configuration documentation [here](https://nvidia-merlin.github.io/HugeCTR/master/hugectr_parameter_server.html)\n",
    "\n",
    "> NOTE: If using the cloud deployment with terraform on AWS, be sure to change the address to the address of the hosted Redis database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "081dbb2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting /model-data/criteo/model/ps.json\n"
     ]
    }
   ],
   "source": [
    "%%writefile $model_folder/ps.json\n",
    "{\n",
    "    \"volatile_db\": {\n",
    "      \"type\": \"redis_cluster\",\n",
    "      \"address\": \"172.20.0.31:6373,172.20.0.32:6374,172.20.0.33:6375\",\n",
    "      \"user_name\":  \"default\",\n",
    "      \"password\": \"\",\n",
    "      \"num_partitions\": 8,\n",
    "      \"allocation_rate\": 268435456,\n",
    "      \"max_get_batch_size\": 10000,\n",
    "      \"max_set_batch_size\": 10000,\n",
    "      \"overflow_margin\": 10000000,\n",
    "      \"overflow_policy\": \"evict_oldest\",\n",
    "      \"overflow_resolution_target\": 0.99,\n",
    "      \"initial_cache_rate\": 1.0,\n",
    "      \"cache_missed_embeddings\": false,\n",
    "      \"update_filters\": [\"^hps_.+$\"]\n",
    "    },\n",
    "    \"supportlonglong\": true,\n",
    "    \"models\":[\n",
    "        {\n",
    "            \"model\":\"dlrm\",\n",
    "            \"sparse_files\":[\"/model-data/criteo/model/dlrm/1/dlrm0_sparse_20000.model\"],\n",
    "            \"dense_file\":\"/model-data/criteo/model/dlrm/1/dlrm_dense_20000.model\",\n",
    "            \"network_file\":\"/model-data/criteo/model/dlrm/1/dlrm.json\",\n",
    "            \"num_of_worker_buffer_in_pool\": 2,\n",
    "            \"num_of_refresher_buffer_in_pool\":1,\n",
    "            \"deployed_device_list\":[0],\n",
    "            \"max_batch_size\":64,\n",
    "            \"default_value_for_each_table\":[0.0,0.0],\n",
    "            \"hit_rate_threshold\":0.5,\n",
    "            \"gpucacheper\":0.1,\n",
    "            \"gpucache\": true,\n",
    "            \"cache_refresh_percentage_per_iteration\":0.0,\n",
    "            \"maxnum_des_feature_per_sample\": 13,\n",
    "            \"maxnum_catfeature_query_per_table_per_sample\":[26],\n",
    "            \"embedding_vecsize_per_table\":[128],\n",
    "            \"slot_num\":26\n",
    "        }\n",
    "    ]  \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d31c9b1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 9268\n",
      "-rw-r--r-- 1 1000 1000    3704 Nov 21 21:08 dlrm.json\n",
      "drwxr-xr-x 2 1000 1000    4096 Nov 21 21:08 dlrm0_sparse_20000.model\n",
      "-rw-r--r-- 1 1000 1000 9479684 Nov 21 21:08 dlrm_dense_20000.model\n",
      "total 8\n",
      "drwxr-xr-x 3 1000 1000 4096 Nov 21 21:08 1\n",
      "-rw-r--r-- 1 1000 1000 1230 Nov 22 00:45 config.pbtxt\n"
     ]
    }
   ],
   "source": [
    "!ls -l $dlrm_version\n",
    "!ls -l $dlrm_model_repo"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0f93bd5",
   "metadata": {},
   "source": [
    "## 3. Start Tritonserver\n",
    "\n",
    "Follow these steps to start triton.\n",
    "\n",
    "First, in another terminal from the one running this jupyterlab instance, run the following command to enter the `triton` container\n",
    "```bash\n",
    "docker exec -it large-scale-recsys_merlin_1 bash\n",
    "```\n",
    "\n",
    "Once inside the container, start the tritonserver with the following command\n",
    "```bash\n",
    "tritonserver --model-repository=/model-data/criteo/model/ --load-model=dlrm --model-control-mode=explicit --backend-directory=/usr/local/hugectr/backends --backend-config=hugectr,ps=/model-data/criteo/model/ps.json  --http-port=8000 --grpc-port=8001 --metrics-port=8002\n",
    "```\n",
    "\n",
    "At this point, you should see triton connecting to Redis and loading in the DLRM model embeddings.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc666eac",
   "metadata": {},
   "source": [
    "## 4. Run Inference with Sample Data "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "319a0f7c",
   "metadata": {},
   "source": [
    "### Check Triton Deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f472f56a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*   Trying 127.0.0.1:8000...\n",
      "* TCP_NODELAY set\n",
      "* Connected to localhost (127.0.0.1) port 8000 (#0)\n",
      "> GET /v2/health/ready HTTP/1.1\n",
      "> Host: localhost:8000\n",
      "> User-Agent: curl/7.68.0\n",
      "> Accept: */*\n",
      "> \n",
      "* Mark bundle as not supporting multiuse\n",
      "< HTTP/1.1 200 OK\n",
      "< Content-Length: 0\n",
      "< Content-Type: text/plain\n",
      "< \n",
      "* Connection #0 to host localhost left intact\n"
     ]
    }
   ],
   "source": [
    "!curl -v localhost:8000/v2/health/ready"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930785f3",
   "metadata": {},
   "source": [
    "### Batchsize=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "12858621",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Successfully read data for 1 stream/streams with 2 step/steps.\n",
      "*** Measurement Settings ***\n",
      "  Batch size: 1\n",
      "  Using \"time_windows\" mode for stabilization\n",
      "  Measurement window: 5000 msec\n",
      "  Using synchronous calls for inference\n",
      "  Stabilizing using average latency\n",
      "\n",
      "Request concurrency: 1\n",
      "  Client: \n",
      "    Request count: 27776\n",
      "    Throughput: 1542.54 infer/sec\n",
      "    Avg latency: 643 usec (standard deviation 227 usec)\n",
      "    p50 latency: 619 usec\n",
      "    p90 latency: 689 usec\n",
      "    p95 latency: 725 usec\n",
      "    p99 latency: 895 usec\n",
      "    Avg HTTP time: 640 usec (send/recv 29 usec + response wait 611 usec)\n",
      "  Server: \n",
      "    Inference count: 27777\n",
      "    Execution count: 27777\n",
      "    Successful request count: 27777\n",
      "    Avg request latency: 524 usec (overhead 1 usec + queue 35 usec + compute input 0 usec + compute infer 488 usec + compute output 0 usec)\n",
      "\n",
      "Inferences/Second vs. Client Average Batch Latency\n",
      "Concurrency: 1, throughput: 1542.54 infer/sec, latency 643 usec\n"
     ]
    }
   ],
   "source": [
    "!perf_analyzer -m dlrm -u localhost:8000 --input-data /model-data/criteo/1.json --shape CATCOLUMN:26 --shape DES:13 --shape ROWINDEX:27"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "150209b7-27d3-4e28-83ad-a028c4268baf",
   "metadata": {},
   "source": [
    "### Batchsize=64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "5e3e4727-9a54-4850-b79a-a3813794541e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Successfully read data for 1 stream/streams with 1 step/steps.\n",
      "*** Measurement Settings ***\n",
      "  Batch size: 1\n",
      "  Using \"time_windows\" mode for stabilization\n",
      "  Measurement window: 5000 msec\n",
      "  Using synchronous calls for inference\n",
      "  Stabilizing using average latency\n",
      "\n",
      "Request concurrency: 1\n",
      "  Client: \n",
      "    Request count: 41139\n",
      "    Throughput: 2285.01 infer/sec\n",
      "    Avg latency: 433 usec (standard deviation 137 usec)\n",
      "    p50 latency: 415 usec\n",
      "    p90 latency: 497 usec\n",
      "    p95 latency: 550 usec\n",
      "    p99 latency: 635 usec\n",
      "    Avg HTTP time: 430 usec (send/recv 28 usec + response wait 402 usec)\n",
      "  Server: \n",
      "    Inference count: 126930\n",
      "    Execution count: 126930\n",
      "    Successful request count: 126930\n",
      "    Avg request latency: 293 usec (overhead 1 usec + queue 42 usec + compute input 0 usec + compute infer 250 usec + compute output 0 usec)\n",
      "\n",
      "Inferences/Second vs. Client Average Batch Latency\n",
      "Concurrency: 1, throughput: 2285.01 infer/sec, latency 433 usec\n"
     ]
    }
   ],
   "source": [
    "!perf_analyzer -m dlrm -u localhost:8000 --input-data /model-data/criteo/64.json --shape CATCOLUMN:1664 --shape DES:832 --shape ROWINDEX:1665"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66e91fa",
   "metadata": {},
   "source": [
    "## 5. Run Inference with the Tritonserver Python Client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "a7bbcb94",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'id': '1', 'model_name': 'dlrm', 'model_version': '1', 'parameters': {'NumSample': 10, 'DeviceID': 0}, 'outputs': [{'name': 'OUTPUT0', 'datatype': 'FP32', 'shape': [10], 'parameters': {'binary_data_size': 40}}]}\n",
      "Prediction Result:\n",
      "[0.01985384 0.02970626 0.02543451 0.02905972 0.08103204 0.02941077\n",
      " 0.02769326 0.0242354  0.02630902 0.02453931]\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from tritonclient.utils import *\n",
    "import tritonclient.http  as httpclient\n",
    "\n",
    "model_name = 'dlrm'\n",
    "CATEGORICAL_COLUMNS=[\"C\" + str(x) for x in range(1, 27)]\n",
    "CONTINUOUS_COLUMNS=[\"I\" + str(x) for x in range(1, 14)]\n",
    "LABEL_COLUMNS = ['label']\n",
    "emb_size_array = [4976199, 25419, 14705, 7112, 19283, 4, 6391, 1282, 60, 3289052, 282487, 138210, 11, 2203, 8901, 67, 4, 948, 15, 5577159, 1385790, 4348882, 178673, 10023, 88, 34]\n",
    "shift = np.insert(np.cumsum(emb_size_array), 0, 0)[:-1]\n",
    "test_df=pd.read_csv(\"/model-data/criteo/infer_test.csv\",sep=',')\n",
    "\n",
    "\n",
    "\n",
    "with httpclient.InferenceServerClient(\"localhost:8000\") as client:\n",
    "    dense_features = np.array([list(test_df.head(10)[CONTINUOUS_COLUMNS].values.flatten())],dtype='float32')\n",
    "    embedding_columns = np.array([list((test_df.head(10)[CATEGORICAL_COLUMNS]+shift).values.flatten())],dtype='int64')\n",
    "    row_ptrs = np.array([list(range(0,261))],dtype='int32')\n",
    "    \n",
    "    inputs = [\n",
    "        httpclient.InferInput(\"DES\", dense_features.shape,\n",
    "                              np_to_triton_dtype(dense_features.dtype)),\n",
    "        httpclient.InferInput(\"CATCOLUMN\", embedding_columns.shape,\n",
    "                              np_to_triton_dtype(embedding_columns.dtype)),\n",
    "        httpclient.InferInput(\"ROWINDEX\", row_ptrs.shape,\n",
    "                              np_to_triton_dtype(row_ptrs.dtype)),\n",
    "\n",
    "    ]\n",
    "\n",
    "    inputs[0].set_data_from_numpy(dense_features)\n",
    "    inputs[1].set_data_from_numpy(embedding_columns)\n",
    "    inputs[2].set_data_from_numpy(row_ptrs)\n",
    "    outputs = [\n",
    "        httpclient.InferRequestedOutput(\"OUTPUT0\")\n",
    "    ]\n",
    "\n",
    "    response = client.infer(model_name,\n",
    "                            inputs,\n",
    "                            request_id=str(1),\n",
    "                            outputs=outputs)\n",
    "\n",
    "    result = response.get_response()\n",
    "    print(result)\n",
    "    print(\"Prediction Result:\")\n",
    "    print(response.as_numpy(\"OUTPUT0\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "01107d9b0f1e8bff803c568b1d711201308d36212c090092c09f341770e357c8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}