speedtest.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## speedtest\n",
    "In this notebook we time how long it takes deepstack to process a set of images. The purpose is to allow comparison of speed on different hosts. We also count the total number of predictions as a quality check\n",
    "\n",
    "```\n",
    "docker run -e VISION-DETECTION=True -e API-KEY=\"\" -v localstorage:/datastore -p 5000:5000 --name deepstack deepquestai/deepstack:cpu-x4-beta\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# deepstack credentials\n",
    "IP_ADDRESS = 'localhost' # 'localhost' # 252=pi4, 139=javier\n",
    "PORT = 80 # 5000\n",
    "API_KEY =  \"\" # if you have not set an api_key, just pass an empty string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepstack.core as ds\n",
    "from pathlib import Path\n",
    "import time\n",
    "import sys\n",
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = list(Path(\"images/\").rglob(\"*.jpg\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Object detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsobject = ds.DeepstackObject(IP_ADDRESS, PORT, API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processed image number 0 : images/scenes/dog.jpg, 1 predictions\n",
      "Processed image number 1 : images/scenes/street.jpg, 8 predictions\n",
      "Processed image number 2 : images/scenes/farm.jpg, 12 predictions\n",
      "Processed image number 3 : images/scenes/pool.jpg, 6 predictions\n",
      "Processed image number 4 : images/scenes/supermarket.jpg, 0 predictions\n",
      "Processed image number 5 : images/scenes/crossing.jpg, 14 predictions\n",
      "Processed image number 6 : images/scenes/porsche.jpg, 1 predictions\n",
      "Processed image number 7 : images/scenes/people_car.jpg, 4 predictions\n",
      "Processed image number 8 : images/scenes/test-image3.jpg, 3 predictions\n",
      "Processed image number 9 : images/scenes/.ipynb_checkpoints/supermarket-checkpoint.jpg, 0 predictions\n",
      "Processed image number 10 : images/faces/me.jpg, 1 predictions\n",
      "Processed image number 11 : images/faces/idris.jpg, 1 predictions\n",
      "Processed image number 12 : images/faces/idris_2.jpg, 2 predictions\n",
      "Processed image number 13 : images/faces/family.jpg, 2 predictions\n",
      "Processed image number 14 : images/faces/face.jpg, 4 predictions\n",
      "Processed image number 15 : images/faces/couple.jpg, 3 predictions\n",
      "Processed image number 16 : images/faces/.ipynb_checkpoints/me-checkpoint.jpg, 1 predictions\n",
      "Processing completes in 6.539522886276245, total of 63 predictions\n",
      "CPU times: user 80.2 ms, sys: 24 ms, total: 104 ms\n",
      "Wall time: 6.54 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "total_predictions = 0 # keep predictions\n",
    "start_time = time.time()\n",
    "\n",
    "for i, img in enumerate(images):\n",
    "\n",
    "    try:\n",
    "        with open(img, 'rb') as image_bytes:\n",
    "            predictions = dsobject.detect(image_bytes)\n",
    "            print(f\"Processed image number {i} : {str(img)}, {len(predictions)} predictions\")\n",
    "            total_predictions = total_predictions + len(predictions)\n",
    "    except Exception as exc:\n",
    "        print(exc)\n",
    "        print(img)\n",
    "\n",
    "end_time = time.time()\n",
    "duration = end_time - start_time\n",
    "print(f\"Processing completes in {duration}, total of {total_predictions} predictions\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## League table\n",
    "| Platform | Speed (sec) | Predictions |\n",
    "| --- | --- | --- |\n",
    "| Pi4 with arm64-2021.06.1 | 15 | 50 |\n",
    "| Jetson xavier with jetpack-2021.06.1 | 5.1 | 50 |\n",
    "| Macbook Pro 2017 cpu-2021.06.1 | 6.5 | 63 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Face detection\n",
    "On Jetson: \n",
    "\n",
    "`sudo docker run --runtime nvidia -d -e VISION-FACE=True -e VISION-DETECTION=True -p 80:5000 deepquestai/deepstack:jetpack-x3-beta`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsface = ds.DeepstackFace(IP_ADDRESS, PORT, API_KEY)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------\n",
      "images/scenes/dog.jpg : 0.01 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/street.jpg : 0.62 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "----------------------\n",
      "images/scenes/farm.jpg : 0.3 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/pool.jpg : 0.28 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/supermarket.jpg : 1.8 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "----------------------\n",
      "images/scenes/crossing.jpg : 0.11 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/porsche.jpg : 0.12 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/people_car.jpg : 0.11 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/test-image3.jpg : 0.21 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/scenes/.ipynb_checkpoints/supermarket-checkpoint.jpg : 1.8 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "----------------------\n",
      "images/faces/me.jpg : 0.81 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "----------------------\n",
      "images/faces/idris.jpg : 0.21 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/faces/idris_2.jpg : 0.06 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/faces/family.jpg : 0.13 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/faces/face.jpg : 0.15 MB\n",
      "Error from Deepstack request, status code: 403\n",
      "----------------------\n",
      "images/faces/couple.jpg : 1.35 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "----------------------\n",
      "images/faces/.ipynb_checkpoints/me-checkpoint.jpg : 0.81 MB\n",
      "Deepstack connection error, check your IP and port: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))\n",
      "Processing completes in 4.5499889850616455, total of 0 faces\n",
      "CPU times: user 127 ms, sys: 45.6 ms, total: 172 ms\n",
      "Wall time: 4.55 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "total_faces = 0 # keep predictions\n",
    "start_time = time.time()\n",
    "\n",
    "for i, img in enumerate(images):\n",
    "    print('----------------------')\n",
    "    try:\n",
    "        img_size = os.path.getsize(img)\n",
    "        img_size = round(img_size/1e6, 2) \n",
    "        print(f\"{img} : {img_size} MB\")\n",
    "        with open(img, 'rb') as image_bytes:\n",
    "            dsface.detect(image_bytes)\n",
    "            predictions = dsface.predictions\n",
    "            print(f\"Processed image number {i} : {str(img)}, {len(predictions)} predictions\")\n",
    "            total_faces = total_faces + len(predictions)\n",
    "    except Exception as exc:\n",
    "        print(exc)\n",
    "        \n",
    "\n",
    "end_time = time.time()\n",
    "duration = end_time - start_time\n",
    "print(f\"Processing completes in {duration}, total of {total_faces} faces\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fire custom model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "dsfire = ds.DeepstackObject(IP_ADDRESS, PORT, API_KEY, min_confidence=0.01, custom_model='fire2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "fire_image_dir = '/Users/robin/Data/fire-dataset/validation/images'\n",
    "fire_images = list(Path(fire_image_dir).rglob(\"*.jpg\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-58fc85387640>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mfire_images\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "fire_images[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "total_predictions = 0 # keep predictions\n",
    "start_time = time.time()\n",
    "\n",
    "for i, img in enumerate(fire_images):\n",
    "\n",
    "    try:\n",
    "        with open(img, 'rb') as image_bytes:\n",
    "            predictions = dsfire.detect(image_bytes)\n",
    "            print(f\"Processed image number {i} : {str(img)}, {len(predictions)} predictions\")\n",
    "            total_predictions = total_predictions + len(predictions)\n",
    "    except Exception as exc:\n",
    "        print(exc)\n",
    "        print(img)\n",
    "\n",
    "end_time = time.time()\n",
    "duration = end_time - start_time\n",
    "print(f\"Processing completes in {duration}, total of {total_predictions} predictions\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}